LoRA+: Efficient Low Rank Adaptation of Large Models

arXiv:2402.12354v1 Announce Type: new
Abstract: In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. …

abstract arxiv cs.ai cs.cl cs.lg embedding finetuning large models leads lora low paper show stat.ml type