Benign overfitting in leaky ReLU networks with moderate input dimension

arXiv:2403.06903v1 Announce Type: new
Abstract: The problem of benign overfitting asks whether it is possible for a model to perfectly fit noisy training data and still generalize well. We study benign overfitting in two-layer leaky ReLU networks trained with the hinge loss on a binary classification task. We consider input data which can be decomposed into the sum of a common signal and a random noise component, which lie on subspaces orthogonal to one another. We characterize conditions on the …

abstract arxiv binary classification cs.lg data hinge layer loss networks overfitting relu stat.ml study training training data type