Non-Vacuous Generalisation Bounds for Shallow Neural Networks. (arXiv:2202.01627v3 [cs.LG] UPDATED)

Web: http://arxiv.org/abs/2202.01627

We focus on a specific class of shallow neural networks with a single hidden
layer, namely those with $L_2$-normalised data and either a sigmoid-shaped
Gaussian error function ("erf") activation or a Gaussian Error Linear Unit
(GELU) activation. For these networks, we derive new generalisation bounds
through the PAC-Bayesian theory; unlike most existing such bounds they apply to
neural networks with deterministic rather than randomised parameters. Our
bounds are empirically non-vacuous when the network is trained with vanilla
stochastic gradient descent …

arxiv lg networks neural neural networks