Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation

arXiv:2403.11808v1 Announce Type: new
Abstract: Existing parameter-efficient fine-tuning (PEFT) methods have achieved significant success on vision transformers (ViTs) adaptation by improving parameter efficiency. However, the exploration of enhancing inference efficiency during adaptation remains underexplored. This limits the broader application of pre-trained ViT models, especially when the model is computationally extensive. In this paper, we propose Dynamic Tuning (DyT), a novel approach to improve both parameter and inference efficiency for ViT adaptation. Specifically, besides using the lightweight adapter modules, we propose …

abstract application arxiv cs.cv dynamic efficiency exploration fine-tuning however inference peft success transformers type vision vision transformers vit