DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. (arXiv:2201.05596v1 [cs.LG])

As the training of giant dense models hits the boundary on the availability
and capability of the hardware resources today, Mixture-of-Experts (MoE) models
become one of the most promising model architectures due to their significant
training cost reduction compared to a quality-equivalent dense model. Its
training cost saving is demonstrated from encoder-decoder models (prior works)
to a 5x saving for auto-aggressive language models (this work along with
parallel explorations). However, due to the much larger model size and unique
architecture, …

ai arxiv moe power training