[R] Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models - Meta AI 2022

Paper: [https://arxiv.org/abs/2208.03306](https://arxiv.org/abs/2208.03306)

Github: [https://github.com/hadasah/btm](https://github.com/hadasah/btm) Code and Models coming soon!

Abstract:

>We present Branch-Train-Merge (BTM), a communication-efficient algorithm for embarrassingly parallel training of large language models (LLMs). We show it is possible to independently train subparts of a new class of LLMs on different subsets of the data, eliminating the massive multi-node synchronization currently required to train LLMs. BTM learns a set of independent expert LMs (ELMs), each specialized to a different textual domain, such as scientific or legal text. These …

ai expert language language models machinelearning meta meta ai training