HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts

arXiv:2402.12656v1 Announce Type: new
Abstract: The Mixture of Experts (MoE) for language models has been proven effective in augmenting the capacity of models by dynamically routing each input token to a specific subset of experts for processing. Despite the success, most existing methods face a challenge for balance between sparsity and the availability of expert knowledge: enhancing performance through increased use of expert knowledge often results in diminishing sparsity during expert selection. To mitigate this contradiction, we propose HyperMoE, a …

abstract arxiv balance capacity challenge cs.ai cs.lg experts face language language models mixture of experts moe processing routing sparsity success token type via