Large Language Models Struggle to Learn Long-Tail Knowledge [R]

[https://arxiv.org/abs/2211.08411](https://arxiv.org/abs/2211.08411)

​

**Abstract:**

The Internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models. However, while certain pieces of information are ubiquitous on the web, others appear extremely rarely. In this paper, we study the relationship between the knowledge memorized by large language models and the information in pre-training datasets scraped from the web. In particular, we show that a language …

abstract code information internet knowledge language language models machinelearning paper relationship study tutorials wealth web