Turn Waste into Worth: Rectifying Top-$k$ Router of MoE

arXiv:2402.12399v1 Announce Type: new
Abstract: Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top-$k$ routing mechanism suffers from redundancy computation and memory costs due to the unbalanced routing. Some experts are overflow, where the exceeding tokens are dropped. While some experts are vacant, which are padded with zeros, negatively impacting model performance. To address the dropped tokens and padding, we propose the Rectify-Router, comprising the Intra-GPU …

abstract arxiv computation computational costs cs.ai cs.lg efficiency experts language language models large language large language models memory mixture of experts moe overflow popular redundancy routing tokens training type waste