Gradient-based Parameter Selection for Efficient Fine-Tuning

arXiv:2312.10136v2 Announce Type: replace
Abstract: With the growing size of pre-trained models, full fine-tuning and storing all the parameters for various downstream tasks is costly and infeasible. In this paper, we propose a new parameter-efficient fine-tuning method, Gradient-based Parameter Selection (GPS), demonstrating that only tuning a few selected parameters from the pre-trained model while keeping the remainder of the model frozen can generate similar or better performance compared with the full model fine-tuning method. Different from the existing popular and …

abstract arxiv cs.cv fine-tuning gps gradient paper parameters pre-trained model pre-trained models tasks type while