Scattered Mixture-of-Experts Implementation

arXiv:2403.08245v1 Announce Type: new
Abstract: We present ScatterMoE, an implementation of Sparse Mixture-of-Experts (SMoE) on GPUs. ScatterMoE builds upon existing implementations, and overcoming some of the limitations to improve inference and training speed, and memory footprint. This implementation achieves this by avoiding padding and making excessive copies of the input.
We introduce ParallelLinear, the main component we use to build our implementation and the various kernels used to speed up the operation. We benchmark our implementation against Megablocks, and show …

abstract arxiv cs.dc cs.lg experts gpus implementation inference limitations making memory padding speed training type