Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models. (arXiv:2110.04478v2 [cs.DC] UPDATED)

Distributed training is a solution to reduce DNN training time by splitting
the task across multiple NPUs (e.g., GPU/TPU). However, distributed training
adds communication overhead between the NPUs in order to synchronize the
gradients and/or activation, depending on the parallelization strategy. In
next-generation platforms for training at scale, NPUs will be connected through
multi-dimensional networks with diverse, heterogeneous bandwidths. This work
identifies a looming challenge of keeping all network dimensions busy and
maximizing the network BW within the hybrid environment …

arxiv distributed dl network policy scheduling training