Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities

arXiv:2403.07585v1 Announce Type: cross
Abstract: The past few years have witnessed the flourishing of large-scale deep neural network models with ever-growing parameter numbers. Training such large-scale models typically requires massive memory and computing resources that exceed those of a single GPU, necessitating distributed training. As GPU performance has rapidly evolved in recent years, computation time has shrunk, thereby increasing the proportion of communication in the overall training time. Therefore, optimizing communication for distributed training has become an urgent issue. In …

abstract advances architecture arxiv communication computing computing resources cs.dc cs.lg deep neural network distributed gpu large-scale models massive memory network neural network numbers opportunities optimization performance resources scale training type