Accelerating Parallel Stochastic Gradient Descent via Non-blocking Mini-batches. (arXiv:2211.00889v1 [cs.LG])

SOTA decentralized SGD algorithms can overcome the bandwidth bottleneck at
the parameter server by using communication collectives like Ring All-Reduce
for synchronization. While the parameter updates in distributed SGD may happen
asynchronously there is still a synchronization barrier to make sure that the
local training epoch at every learner is complete before the learners can
advance to the next epoch. The delays in waiting for the slowest
learners(stragglers) remain to be a problem in the synchronization steps of
these state-of-the-art …

arxiv blocking gradient stochastic