Global Momentum Compression for Sparse Communication in Distributed Learning

arXiv:1905.12948v3 Announce Type: replace-cross
Abstract: With the rapid growth of data, distributed momentum stochastic gradient descent~(DMSGD) has been widely used in distributed learning, especially for training large-scale deep models. Due to the latency and limited bandwidth of the network, communication has become the bottleneck of distributed learning. Communication compression with sparsified gradient, abbreviated as \emph{sparse communication}, has been widely employed to reduce communication cost. All existing works about sparse communication in DMSGD employ local momentum, in which the momentum only …

abstract arxiv bandwidth become communication compression cs.lg data distributed distributed learning global gradient growth latency network scale stat.ml stochastic training type