Asymmetry in Low-Rank Adapters of Foundation Models

arXiv:2402.16842v2 Announce Type: replace
Abstract: Parameter-efficient fine-tuning optimizes large, pre-trained foundation models by updating a subset of parameters; in this class, Low-Rank Adaptation (LoRA) is particularly effective. Inspired by an effort to investigate the different roles of LoRA matrices during fine-tuning, this paper characterizes and leverages unexpected asymmetry in the importance of low-rank adapter matrices. Specifically, when updating the parameter matrices of a neural network by adding a product $BA$, we observe that the $B$ and $A$ matrices have distinct …

abstract arxiv class cs.lg fine-tuning foundation importance lora low low-rank adaptation paper parameters roles type