AB-Training: A Communication-Efficient Approach for Distributed Low-Rank Learning

arXiv:2405.01067v1 Announce Type: new
Abstract: Communication bottlenecks hinder the scalability of distributed neural network training, particularly on distributed-memory computing clusters. To significantly reduce this communication overhead, we introduce AB-training, a novel data-parallel training method that decomposes weight matrices into low-rank representations and utilizes independent group-based training. This approach consistently reduces network traffic by 50% across multiple scaling scenarios, increasing the training potential on communication-constrained systems. Our method exhibits regularization effects at smaller scales, leading to improved generalization for models like …

abstract arxiv bottlenecks communication computing cs.ai cs.dc cs.lg data distributed hinder independent low memory network network training neural network novel reduce scalability traffic training type