Adaptive Worker Grouping For Communication-Efficient and Straggler-Tolerant Distributed SGD. (arXiv:2201.04301v1 [cs.IT])

Wall-clock convergence time and communication load are key performance
metrics for the distributed implementation of stochastic gradient descent (SGD)
in parameter server settings. Communication-adaptive distributed Adam (CADA)
has been recently proposed as a way to reduce communication load via the
adaptive selection of workers. CADA is subject to performance degradation in
terms of wall-clock convergence time in the presence of stragglers. This paper
proposes a novel scheme named grouping-based CADA (G-CADA) that retains the
advantages of CADA in reducing the …

arxiv communication distributed