AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodes

arXiv:2404.09679v1 Announce Type: cross
Abstract: Many distributed training techniques like Parameter Server and AllReduce have been proposed to take advantage of the increasingly large data and rich features. However, stragglers frequently occur in distributed training due to resource contention and hardware heterogeneity, which significantly hampers the training efficiency. Previous works only address part of the stragglers and could not adaptively solve various stragglers in practice. Additionally, it is challenging to use a systematic framework to address all stragglers because different …

abstract arxiv cs.dc cs.lg data distributed efficiency features framework hardware however leader nodes server training type