Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective. (arXiv:2110.06256v2 [cs.LG] UPDATED)

Web: http://arxiv.org/abs/2110.06256

This work examines the deep disconnect between existing theoretical analyses
of gradient-based algorithms and the practice of training deep neural networks.
Specifically, we provide numerical evidence that in large-scale neural network
training (e.g., ImageNet + ResNet101, and WT103 + TransformerXL models), the
neural network's weights do not converge to stationary points where the
gradient of the loss is zero. Remarkably, however, we observe that even though
the weights do not converge to stationary points, the progress in minimizing
the loss …

arxiv lg network neural neural network perspective