Sparse is Enough in Fine-tuning Pre-trained Large Language Models

arXiv:2312.11875v2 Announce Type: replace-cross
Abstract: With the prevalence of pre-training-fine-tuning paradigm, how to efficiently adapt the pre-trained model to the downstream tasks has been an intriguing issue. Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed for low-cost adaptation. Although PEFT has demonstrated effectiveness and been widely applied, the underlying principles are still unclear. In this paper, we adopt the PAC-Bayesian generalization error bound, viewing pre-training as a shift of prior distribution which leads to a tighter bound for generalization error. We …

arxiv cs.ai cs.cl cs.lg fine-tuning language language models large language large language models type