Scaling laws for learning with real and surrogate data

Collecting large quantities of high-quality data is often prohibitively expensive or impractical, and a crucial bottleneck in machine learning. One may instead augment a small set of $n$ data points from the target distribution with data from more accessible sources like public datasets, data collected under different circumstances, or synthesized by generative models. Blurring distinctions, we refer to such data as `surrogate data'.
We define a simple scheme for integrating surrogate data into training and use both theoretical models and …

cs.ai cs.lg data datasets distribution laws machine machine learning public quality quality data scaling set small stat.ml synthesized