The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks. (arXiv:2108.11489v2 [stat.ML] UPDATED)

The recent success of neural network models has shone light on a rather
surprising statistical phenomenon: statistical models that perfectly fit noisy
data can generalize well to unseen test data. Understanding this phenomenon of
$\textit{benign overfitting}$ has attracted intense theoretical and empirical
study. In this paper, we consider interpolating two-layer linear neural
networks trained with gradient flow on the squared loss and derive bounds on
the excess risk when the covariates satisfy sub-Gaussianity and
anti-concentration properties, and the noise is …

arxiv bias linear ml networks overfitting