MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning

arXiv:2402.11260v1 Announce Type: new
Abstract: Adapting large language models (LLMs) to new domains/tasks and enabling them to be efficient lifelong learners is a pivotal challenge. In this paper, we propose MoRAL, i.e., Mixture-of-Experts augmented Low-Rank Adaptation for Lifelong Learning. MoRAL combines the multi-tasking abilities of MoE with the fine-tuning abilities of LoRA for effective life-long learning of LLMs. In contrast to the conventional approaches that use factual triplets as inputs MoRAL relies on simple question-answer pairs, which is a more …

abstract arxiv challenge cs.ai cs.cl domains enabling experts fine-tuning language language models large language large language models lifelong learning llms lora low low-rank adaptation moe paper pivotal tasks them type