[R] A Tale of Tails: Model Collapse as a Change of Scaling Laws

**Paper**: [https://arxiv.org/abs/2402.07043](https://arxiv.org/abs/2402.07043)

**Abstract**:

>As AI model size grows, neural *scaling laws* have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natural) training data. Yet, the widespread use of popular models means that the ecosystem of online data and text will co-evolve to progressively contain increased amounts of synthesized data. In this paper we ask: *How will the scaling laws change in the inevitable regime where synthetic data …

abstract ai model become capacity data ecosystem human improvements large models laws machinelearning natural popular scaling synthesized text tool training training data will