Near-Optimal Sparse Allreduce for Distributed Deep Learning. (arXiv:2201.07598v1 [cs.DC])

Communication overhead is one of the major obstacles to train large deep
learning models at scale. Gradient sparsification is a promising technique to
reduce the communication volume. However, it is very challenging to obtain real
performance improvement because of (1) the difficulty of achieving an scalable
and efficient sparse allreduce algorithm and (2) the sparsification overhead.
This paper proposes O$k$-Top$k$, a scheme for distributed training with sparse
gradients. O$k$-Top$k$ integrates a novel sparse allreduce algorithm (less than
6$k$ communication volume …

arxiv deep learning distributed learning