COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Training

arXiv:2211.16648v2 Announce Type: replace-cross
Abstract: Modern Deep Learning (DL) models have grown to sizes requiring massive clusters of specialized, high-end nodes to train. Designing such clusters to maximize both performance and utilization--to amortize their steep cost--is a challenging task requiring careful balance of compute, memory, and network resources. Moreover, a plethora of each model's tuning knobs drastically affect the performance, with optimal values often depending on the underlying cluster's characteristics, which necessitates a complex cluster-workload co-design process. To facilitate the …

abstract arxiv balance cluster comet compute cost cs.ai cs.dc cs.lg deep learning deep learning training design designing distributed massive memory methodology modern network nodes performance resources train training type