Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning

arXiv:2402.17263v1 Announce Type: new
Abstract: Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models' scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to full-parameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters …

abstract arxiv cs.cl diversity ensemble fine-tuning language language models large language large language models llms lora low low-rank adaptation peft popular process scale tasks type