Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

arXiv:2403.07816v1 Announce Type: new
Abstract: We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts from a seed model, which is branched to train experts in embarrassingly parallel fashion with high throughput and reduced communication cost. After individual experts are asynchronously trained, BTX brings together their feedforward parameters as experts in Mixture-of-Expert (MoE) layers and averages the remaining parameters, …

abstract arxiv capabilities coding cs.ai cs.cl domains expert experts fashion knowledge language language models large language large language models llm llms math multiple reasoning seed train training type world