Unlocking the Global Synergies in Low-Rank Adapters

arXiv:2406.14956v1 Announce Type: cross
Abstract: Low-rank Adaption (LoRA) has been the de-facto parameter-efficient fine-tuning technique for large language models. We present HeteroLoRA, a light-weight search algorithm that leverages zero-cost proxies to allocate the limited LoRA trainable parameters across the model for better fine-tuned performance. In addition to the allocation for the standard LoRA-adapted models, we also demonstrate the efficacy of HeteroLoRA by performing the allocation in a more challenging search space that includes LoRA modules and LoRA-adapted shortcut connections. Experiments …

abstract algorithm arxiv cost cs.cl cs.lg fine-tuning global language language models large language large language models light lora low parameters performance proxies search standard tuning type