Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms

arXiv:2404.12674v1 Announce Type: cross
Abstract: Characterizing and predicting the training performance of modern machine learning (ML) workloads on compute systems with compute and communication spread between CPUs, GPUs, and network devices is not only the key to optimization and planning but also a complex goal to achieve. The primary challenges include the complexity of synchronization and load balancing between CPUs and GPUs, the variance in input data distribution, and the use of different communication devices and topologies (e.g., NVLink, PCIe, …

abstract arxiv communication compute cpus cs.dc cs.lg cs.pf devices gpu gpus key machine machine learning modeling modern multi-gpu network optimization performance planning platforms systems the key training type universal workloads