LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild

arXiv:2402.09997v1 Announce Type: cross
Abstract: Low-Rank Adaptation (LoRA) provides an effective yet efficient solution for fine-tuning large language models (LLM). The modular and plug-and-play nature of LoRA enables the integration of diverse domain-specific LoRAs to enhance the capabilities of LLMs. Previous research on exploiting multiple LoRAs either focuses on specific isolated downstream tasks or fixes the selection of LoRAs during training. However, in real-world scenarios, LLMs receive diverse prompts covering different tasks, and the pool of candidate LoRAs is often …

abstract arxiv capabilities cs.ai cs.cl cs.lg diverse domain fine-tuning integration language language models large language large language models llm llms lora low low-rank adaptation mixed modular multiple nature research retrieval solution tasks type