Normalization effects on shallow neural networks and related asymptotic expansions. (arXiv:2011.10487v3 [stat.ML] UPDATED)

We consider shallow (single hidden layer) neural networks and characterize
their performance when trained with stochastic gradient descent as the number
of hidden units $N$ and gradient descent steps grow to infinity. In particular,
we investigate the effect of different scaling schemes, which lead to different
normalizations of the neural network, on the network's statistical output,
closing the gap between the $1/\sqrt{N}$ and the mean-field $1/N$
normalization. We develop an asymptotic expansion for the neural network's
statistical output pointwise with …

arxiv effects ml networks neural networks normalization