Characterization of Large Language Model Development in the Datacenter

arXiv:2403.07648v1 Announce Type: cross
Abstract: Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs, often riddled with numerous challenges such as frequent hardware failures, intricate parallelization strategies, and imbalanced resource utilization. In this paper, we present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme. Specifically, we investigate discrepancies between LLMs and prior task-specific Deep Learning …

abstract arxiv challenges cluster cs.dc cs.lg datacenter development hardware however language language model language models large language large language model large language models llms model development paper parallelization performance resources scale strategies tasks type