Scaling Language Models: Methods, Analysis & Insights from Training Gopher. (arXiv:2112.11446v2 [cs.CL] UPDATED)

Web: http://arxiv.org/abs/2112.11446

Language modelling provides a step towards intelligent communication systems
by harnessing large repositories of written human knowledge to better predict
and understand the world. In this paper, we present an analysis of
Transformer-based language model performance across a wide range of model
scales -- from models with tens of millions of parameters up to a 280 billion
parameter model called Gopher. These models are evaluated on 152 diverse tasks,
achieving state-of-the-art performance across the majority. Gains from scale
are largest …

analysis arxiv insights language language models models scaling training