MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

arXiv:2402.15627v1 Announce Type: new
Abstract: We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high …

abstract arxiv building challenges cs.dc cs.lg design efficiency engineering experience gpus implementation language language model language models language model training large language large language model large language models llms production scale scaling stability training training llms type