Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

arXiv:2404.01413v1 Announce Type: new
Abstract: The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs? Recent investigations into model-data feedback loops discovered that such loops can lead to model collapse, a phenomenon where performance progressively degrades with each model-fitting iteration until the latest model becomes useless. However, several recent papers studying model collapse assumed that new data replace old data over time rather …

abstract arxiv breaking cs.ai cs.cl cs.et cs.lg data feedback generated generative generative models investigations model collapse pretraining question raises recursion scale stat.ml synthetic synthetic data type web