Toward Inference-optimal Mixture-of-Expert Large Language Models

arXiv:2404.02852v1 Announce Type: new
Abstract: Mixture-of-Expert (MoE) based large language models (LLMs), such as the recent Mixtral and DeepSeek-MoE, have shown great promise in scaling model size without suffering from the quadratic growth of training cost of dense transformers. Like dense models, training MoEs requires answering the same question: given a training budget, what is the optimal allocation on the model size and number of tokens? We study the scaling law of MoE-based LLMs regarding the relations between the model …

abstract arxiv budget cost cs.lg deepseek expert growth inference language language models large language large language models llms mixtral moe question scaling training transformers type