LoRA Dropout as a Sparsity Regularizer for Overfitting Control

arXiv:2404.09610v1 Announce Type: new
Abstract: Parameter-efficient fine-tuning methods, represented by LoRA, play an essential role in adapting large-scale pre-trained models to downstream tasks. However, fine-tuning LoRA-series models also faces the risk of overfitting on the training dataset, and yet there's still a lack of theoretical guidance and practical mechanism to control overfitting on LoRA-based PEFT methods. In this paper, we propose a LoRA Dropout mechanism for the LoRA-based methods by introducing random noises to the learnable low-rank matrices and increasing …

abstract arxiv control cs.ai cs.lg dataset dropout fine-tuning guidance however lora overfitting practical pre-trained models risk role scale series sparsity tasks training type