RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning. (arXiv:2211.00839v1 [cs.LG])

The convergence of SGD based distributed training algorithms is tied to the
data distribution across workers. Standard partitioning techniques try to
achieve equal-sized partitions with per-class population distribution in
proportion to the total dataset. Partitions having the same overall population
size or even the same number of samples per class may still have Non-IID
distribution in the feature space. In heterogeneous computing environments,
when devices have different computing capabilities, even-sized partitions
across devices can lead to the straggler problem in …

arxiv distributed environment partitioning