PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization

arXiv:2402.16141v1 Announce Type: new
Abstract: Supervised fine-tuning is the most common method to adapt large language models (LLMs) to downstream tasks, but full fine-tuning LLMs requires massive computational resources. Recently, parameter-efficient fine-tuning (PEFT) methods have been widely studied due to its cost-effectiveness. LoRA is one of the most widely used methods, which assumes that the optimization process is essentially low-dimensional. Although LoRA fine-tuning is effective, there is still a performance gap compared to full fine-tuning, since its weight update is …

abstract adapt arxiv breaking computational cost cs.cl fine-tuning language language models large language large language models llms lora low massive optimization peft resources supervised fine-tuning tasks type