Normalization effects on deep neural networks. (arXiv:2209.01018v1 [cs.LG])

We study the effect of normalization on the layers of deep neural networks of
feed-forward type. A given layer $i$ with $N_{i}$ hidden units is allowed to be
normalized by $1/N_{i}^{\gamma_{i}}$ with $\gamma_{i}\in[1/2,1]$ and we study
the effect of the choice of the $\gamma_{i}$ on the statistical behavior of the
neural network's output (such as variance) as well as on the test accuracy on
the MNIST data set. We find that in terms of variance of the neural network's
output …

arxiv effects networks neural networks normalization