Noisy Interpolation Learning with Shallow Univariate ReLU Networks

arXiv:2307.15396v3 Announce Type: replace
Abstract: Understanding how overparameterized neural networks generalize despite perfect interpolation of noisy training data is a fundamental question. Mallinar et. al. 2022 noted that neural networks seem to often exhibit ``tempered overfitting'', wherein the population risk does not converge to the Bayes optimal error, but neither does it approach infinity, yielding non-trivial generalization. However, this has not been studied rigorously. We provide the first rigorous analysis of the overfitting behavior of regression with minimum norm ($\ell_2$ …

abstract arxiv bayes converge cs.lg data error networks neural networks overfitting population question relu risk training training data type understanding