Harder Tasks Need More Experts: Dynamic Routing in MoE Models

arXiv:2403.07652v1 Announce Type: new
Abstract: In this paper, we introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models, aiming to enhance computational efficiency and model performance by adjusting the number of activated experts based on input difficulty. Unlike traditional MoE approaches that rely on fixed Top-K routing, which activates a predetermined number of experts regardless of the input's complexity, our method dynamically selects experts based on the confidence level in expert selection for each input. This …

arxiv cs.cl cs.lg dynamic experts moe routing tasks type