Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts

arXiv:2404.05019v1 Announce Type: new
Abstract: Expert parallelism has been introduced as a strategy to distribute the computational workload of sparsely-gated mixture-of-experts (MoE) models across multiple computing devices, facilitating the execution of these increasingly large-scale models. However, the All-to-All communication intrinsic to expert parallelism constitutes a significant overhead, diminishing the MoE models' efficiency. Current optimization approaches offer some relief, yet they are constrained by the sequential interdependence of communication and computation operations. To address this limitation, we present a novel shortcut-connected …

abstract arxiv communication computational computing cs.cl cs.dc cs.lg current devices efficiency expert experts however intrinsic large-scale models moe multiple optimization scale shortcut strategy type