Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing

arXiv:2404.16914v1 Announce Type: new
Abstract: MoE facilitates the development of large models by making the computational complexity of the model no longer scale linearly with increasing parameters. The learning sparse gating network selects a set of experts for each token to be processed; however, this may lead to differences in the number of tokens processed by each expert over several successive iterations, i.e., the expert load fluctuations, which reduces computational parallelization and resource utilization. To this end, we traced and …

abstract arxiv complexity computational cs.ai cs.cl cs.lg development distribution expert experts however large models making moe network parameters prediction scale set token type