Scaling Limit of Neural Networks with the Xavier Initialization and Convergence to a Global Minimum. (arXiv:1907.04108v3 [math.PR] UPDATED)

We analyze single-layer neural networks with the Xavier initialization in the
asymptotic regime of large numbers of hidden units and large numbers of
stochastic gradient descent training steps. The evolution of the neural network
during training can be viewed as a stochastic system and, using techniques from
stochastic analysis, we prove the neural network converges in distribution to a
random ODE with a Gaussian distribution. The limit is completely different than
in the typical mean-field results for neural networks due …

arxiv convergence global math networks neural networks pr scaling