Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping

arXiv:2404.19429v1 Announce Type: cross
Abstract: The Mixture-of-Expert (MoE) technique plays a crucial role in expanding the size of DNN model parameters. However, it faces the challenge of extended all-to-all communication latency during the training process. Existing methods attempt to mitigate this issue by overlapping all-to-all with expert computation. Yet, these methods frequently fall short of achieving sufficient overlap, consequently restricting the potential for performance enhancements. In our study, we extend the scope of this challenge by considering overlap at the …

abstract arxiv challenge communication computation cs.dc cs.lg dnn expert experts graph however issue latency moe parameters process role training type via