Mixture of LoRA Experts

arXiv:2404.13628v1 Announce Type: cross
Abstract: LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques. Due to the modular nature of LoRA's plug-and-play plugins, researchers have delved into the amalgamation of multiple LoRAs to empower models to excel across various downstream tasks. Nonetheless, extant approaches for LoRA fusion grapple with inherent challenges. …

abstract array arxiv cs.cl cs.lg cs.mm diverse efficiency experts fine-tuning lora modular nature plugins pre-trained models researchers tasks type