Memorize to Generalize: on the Necessity of Interpolation in High Dimensional Linear Regression. (arXiv:2202.09889v2 [stat.ML] UPDATED)

Web: http://arxiv.org/abs/2202.09889

We examine the necessity of interpolation in overparameterized models, that
is, when achieving optimal predictive risk in machine learning problems
requires (nearly) interpolating the training data. In particular, we consider
simple overparameterized linear regression $y = X \theta + w$ with random
design $X \in \mathbb{R}^{n \times d}$ under the proportional asymptotics $d/n
\to \gamma \in (1, \infty)$. We precisely characterize how prediction (test)
error necessarily scales with training error in this setting. An implication of
this characterization is that …

arxiv linear linear regression ml on regression