ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

arXiv:2403.16187v1 Announce Type: new
Abstract: Parameter-efficient fine-tuning (PEFT) is widely studied for its effectiveness and efficiency in the era of large language models. Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular and representative method. However, it is implemented with a fixed intrinsic rank that might not be the ideal setting for the downstream tasks. Recognizing the need for more flexible downstream task adaptation, we extend the methodology of LoRA to an innovative approach we call allocating low-rank adaptation …

abstract arxiv cs.cl efficiency fine-tuning however intrinsic language language models large language large language models lora low low-rank adaptation peft performance popular type