The effect of Leaky ReLUs on the training and generalization of overparameterized networks

arXiv:2402.11942v1 Announce Type: new
Abstract: We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, $\alpha$. We show that $\alpha =-1$, which corresponds to the absolute value activation function, is optimal for the training …

abstract arxiv class convergence cs.lg error errors functions linear networks neural networks nns rate relu training type