A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs

arXiv:2305.13525v2 Announce Type: replace
Abstract: Large communication costs are a critical bottleneck in training state-of-the-art neural networks on distributed systems. This paper introduces AxoNN, a novel four-dimensional (4D) parallelization approach, inspired by Agarwal's algorithm for matrix multiplication, for parallelizing tensor computations in deep learning, AxoNN employs two key strategies to minimize communication overhead. First, we optimize communication by overlapping expensive collective operations (reduce-scatter, all-gather, all-reduce) with computations. Our experiments with a 20-billion parameter transformer model demonstrate that these optimizations deliver …

abstract algorithm art arxiv communication costs cs.ai cs.dc cs.lg cs.pf deep learning distributed distributed systems gpus hybrid key matrix matrix multiplication networks neural networks novel paper parallelization scale state systems tensor training type