SNP: Structured Neuron-level Pruning to Preserve Attention Scores

arXiv:2404.11630v1 Announce Type: new
Abstract: Multi-head self-attention (MSA) is a key component of Vision Transformers (ViTs), which have achieved great success in various vision tasks. However, their high computational cost and memory footprint hinder their deployment on resource-constrained devices. Conventional pruning approaches can only compress and accelerate the MSA module using head pruning, although the head is not an atomic unit. To address this issue, we propose a novel graph-aware neuron-level pruning method, Structured Neuron-level Pruning (SNP). SNP prunes neurons …

abstract arxiv attention computational cost cs.ai cs.cv deployment devices head hinder however key memory multi-head neuron pruning self-attention success tasks transformers type vision vision transformers