Towards an empirical understanding of MoE design choices

arXiv:2402.13089v1 Announce Type: new
Abstract: In this study, we systematically evaluate the impact of common design choices in Mixture of Experts (MoEs) on validation performance, uncovering distinct influences at token and sequence levels. We also present empirical evidence showing comparable performance between a learned router and a frozen, randomly initialized router, suggesting that learned routing may not be essential. Our study further reveals that Sequence-level routing can result in topic-specific weak expert specialization, in contrast to syntax specialization observed with …

abstract arxiv cs.ai cs.cl cs.lg design evidence experts impact mixture of experts moe performance study token type understanding validation