LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

arXiv:2312.09979v3 Announce Type: replace
Abstract: Supervised fine-tuning (SFT) is a crucial step for large language models (LLMs), enabling them to align with human instructions and enhance their capabilities in downstream tasks. Increasing instruction data substantially is a direct solution to align the model with a broader range of downstream tasks or notably improve its performance on a specific task. However, we find that large-scale increases in instruction data can damage the world knowledge previously stored in LLMs. To address this …

abstract arxiv capabilities cs.cl data enabling fine-tuning human knowledge language language models large language large language models llms moe plugin sft solution style supervised fine-tuning tasks them type via world