A Quadratic Synchronization Rule for Distributed Deep Learning

arXiv:2310.14423v2 Announce Type: replace
Abstract: In distributed deep learning with data parallelism, synchronizing gradients at each training step can cause a huge communication overhead, especially when many nodes work together to train large models. Local gradient methods, such as Local SGD, address this issue by allowing workers to compute locally for $H$ steps without synchronizing with others, hence reducing communication frequency. While $H$ has been viewed as a hyperparameter to trade optimization efficiency for communication cost, recent research indicates that …

abstract arxiv communication compute cs.lg data deep learning distributed gradient issue large models nodes synchronization together train training type work workers