LoRA Meets Dropout under a Unified Framework

arXiv:2403.00812v1 Announce Type: new
Abstract: With the remarkable capabilities, large language models (LLMs) have emerged as essential elements in numerous NLP applications, while parameter-efficient finetuning, especially LoRA, has gained popularity as a lightweight approach for model customization. Meanwhile, various dropout methods, initially designed for full finetuning with all the parameters updated, alleviates overfitting associated with excessive parameter redundancy. Hence, a possible contradiction arises from negligible trainable parameters of LoRA and the effectiveness of previous dropout methods, which has been largely …

abstract applications arxiv capabilities cs.ai cs.cl customization dropout finetuning framework language language models large language large language models llms lora model customization nlp overfitting parameters type