The Expressive Power of Low-Rank Adaptation

arXiv:2310.17513v3 Announce Type: replace
Abstract: Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that leverages low-rank adaptation of weight matrices, has emerged as a prevalent technique for fine-tuning pre-trained models such as large language models and diffusion models. Despite its huge success in practice, the theoretical underpinnings of LoRA have largely remained unexplored. This paper takes the first step to bridge this gap by theoretically analyzing the expressive power of LoRA. We prove that, for fully connected neural networks, LoRA can …

abstract arxiv cs.ai cs.cl cs.lg diffusion diffusion models fine-tuning language language models large language large language models lora low low-rank adaptation paper power practice pre-trained models stat.ml success type