DeepMind’s DiLoCo Revolutionizes Language Model Training with 500× Less Communication

In a new paper DiLoCo: Distributed Low-Communication Training of Language Models, a Google DeepMind research team presents Distributed Low-Communication (DiLoCo). DiLoCo employs a distributed optimization algorithm that facilitates the training of language models on islands of poorly connected devices, surpassing the performance of fully synchronous models while reducing communication by 500 times.

ai algorithm artificial intelligence communication connected devices deepmind deepmind research deep-neural-networks devices distributed google google deepmind language language model language models large language model low machine learning machine learning & data science ml optimization paper performance research research team team technology training