From Tempered to Benign Overfitting in ReLU Neural Networks

arXiv:2305.15141v3 Announce Type: replace
Abstract: Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. This phenomenon motivated a large body of work on "benign overfitting", where interpolating predictors achieve near-optimal performance. Recently, it was conjectured and empirically observed that the behavior of NNs is often better described as "tempered overfitting", where the performance is non-optimal yet also non-trivial, and degrades as a function of the noise level. However, a theoretical justification of …

abstract arxiv behavior cs.lg cs.ne data near networks neural networks nns overfitting performance relu stat.ml type work