[R] Scaling Data-Constrained Language Models - Hugging Face et al. 2023

Paper: [https://arxiv.org/abs/2305.16264](https://arxiv.org/abs/2305.16264)

GitHub: [https://github.com/huggingface/datablations](https://github.com/huggingface/datablations)

License:

>All models & code are licensed under Apache 2.0. Filtered datasets are released with the same license as the datasets they stem from.

Abstract:

>The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, …

abstract apache code datasets license machinelearning stem