Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices. (arXiv:2103.03239v4 [cs.LG] UPDATED)

Training deep neural networks on large datasets can often be accelerated by
using multiple compute nodes. This approach, known as distributed training, can
utilize hundreds of computers via specialized message-passing protocols such as
Ring All-Reduce. However, running these protocols at scale requires reliable
high-speed networking that is only available in dedicated clusters. In
contrast, many real-world applications, such as federated learning and
cloud-based distributed training, operate on unreliable devices with unstable
network bandwidth. As a result, these applications are restricted …

arxiv communication devices training