Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach

arXiv:2403.19067v1 Announce Type: new
Abstract: Parameter-efficient fine-tuning for pre-trained Vision Transformers aims to adeptly tailor a model to downstream tasks by learning a minimal set of new adaptation parameters while preserving the frozen majority of pre-trained parameters. Striking a balance between retaining the generalizable representation capacity of the pre-trained model and acquiring task-specific features poses a key challenge. Currently, there is a lack of focus on guiding this delicate trade-off. In this study, we approach the problem from the perspective …

arxiv cs.cv design fine-tuning low residual transformer type vision