DoRA: Weight-Decomposed Low-Rank Adaptation

arXiv:2402.09353v1 Announce Type: new
Abstract: Among the widely used parameter-efficient finetuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs. However, there still often exists an accuracy gap between these methods and full fine-tuning (FT). In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed LowRank Adaptation (DoRA). DoRA …

abstract accuracy analysis arxiv costs cs.cl cs.cv fine-tuning finetuning gap inference inference costs lora low low-rank adaptation novel peft type variants work