SparDL: Distributed Deep Learning Training with Efficient Sparse Communication

arXiv:2304.00737v2 Announce Type: replace
Abstract: Top-k sparsification has recently been widely used to reduce the communication volume in distributed deep learning. However, due to the Sparse Gradient Accumulation (SGA) dilemma, the performance of top-k sparsification still has limitations. Recently, a few methods have been put forward to handle the SGA dilemma. Regrettably, even the state-of-the-art method suffers from several drawbacks, e.g., it relies on an inefficient communication algorithm and requires extra transmission steps. Motivated by the limitations of existing methods, …

abstract arxiv communication cs.dc cs.lg deep learning deep learning training distributed gradient gradient accumulation limitations performance reduce training type