CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks

Distributed training of foundation models, especially large language models
(LLMs), is communication-intensive and so has heavily relied on centralized
data centers with fast interconnects. Can we train on slow networks and
unlock the potential of decentralized infrastructure for foundation models?
In this paper, we propose CocktailSGD, a novel communication-efficient
training framework that combines three distinct compression techniques --
random sparsification, top-K sparsification, and quantization -- to achieve
much greater compression than each individual technique alone. We justify
the benefit of …

centralized data communication data data centers decentralized distributed fine-tuning foundation framework infrastructure interconnects language language models large language large language models llms networks novel paper research training