GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. (arXiv:2112.06905v2 [cs.CL] UPDATED)

Scaling language models with more data, compute and parameters has driven
significant progress in natural language processing. For example, thanks to
scaling, GPT-3 was able to achieve strong results on in-context learning tasks.
However, training these large dense models requires significant amounts of
computing resources. In this paper, we propose and develop a family of language
models named GLaM (Generalist Language Model), which uses a sparsely activated
mixture-of-experts architecture to scale the model capacity while also
incurring substantially less training …

arxiv experts glam language language models scaling