Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment

arXiv:2312.03549v4 Announce Type: replace
Abstract: Large language models (LLMs) such as GPT-3, OPT, and LLaMA have demonstrated remarkable accuracy in a wide range of tasks. However, training these models can incur significant expenses, often requiring tens of thousands of GPUs for months of continuous operation. Typically, this training is carried out in specialized GPU clusters equipped with homogeneous high-speed Remote Direct Memory Access (RDMA) network interface cards (NICs). The acquisition and maintenance of such dedicated clusters is challenging. Current LLM …

abstract accuracy arxiv continuous cs.cl cs.dc distributed environment gpt gpt-3 gpus however language language models large language large language models llama llms tasks training type