HEAT: Head-level Parameter Efficient Adaptation of Vision Transformers with Taylor-expansion Importance Scores

arXiv:2404.08894v1 Announce Type: cross
Abstract: Prior computer vision research extensively explores adapting pre-trained vision transformers (ViT) to downstream tasks. However, the substantial number of parameters requiring adaptation has led to a focus on Parameter Efficient Transfer Learning (PETL) as an approach to efficiently adapt large pre-trained models by training only a subset of parameters, achieving both parameter and storage efficiency. Although the significantly reduced parameters have shown promising performance under transfer learning scenarios, the structural redundancy inherent in the model …

abstract adapt arxiv computer computer vision cs.cv cs.lg expansion focus head heat however importance parameters pre-trained models prior research tasks taylor transfer transfer learning transformers type vision vision research vision transformers vit