TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learning

arXiv:2304.05301v2 Announce Type: replace-cross
Abstract: The surge of artificial intelligence, specifically large language models, has led to a rapid advent towards the development of large-scale machine learning training clusters. Collective communications within these clusters tend to be heavily bandwidth-bound, necessitating techniques to optimally utilize the available network bandwidth. This puts the routing algorithm for the collective at the forefront of determining the performance. Unfortunately, communication libraries used in distributed machine learning today are limited by a fixed set of routing …

abstract algorithm artificial artificial intelligence arxiv bandwidth collective communications cs.dc cs.lg development distributed intelligence language language models large language large language models machine machine learning network scale topology training type