MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts

arXiv:2404.15159v1 Announce Type: new
Abstract: Large Language Models (LLMs) have showcased exceptional performance across a wide array of Natural Language Processing (NLP) tasks. Fine-tuning techniques are commonly utilized to tailor pre-trained models to specific applications. While methods like LoRA have effectively tackled GPU memory constraints during fine-tuning, their applicability is often restricted to limited performance, especially on multi-task. On the other hand, Mix-of-Expert (MoE) models, such as Mixtral 8x7B, demonstrate remarkable performance across multiple NLP tasks while maintaining a reduced …

abstract applications array arxiv constraints cs.ai cs.cl experts fine-tuning gpu language language models language processing large language large language models llms lora memory mixture of experts natural natural language natural language processing nlp performance pre-trained models processing tasks type