Regularized Gauss-Newton for Optimizing Overparameterized Neural Networks

arXiv:2404.14875v1 Announce Type: new
Abstract: The generalized Gauss-Newton (GGN) optimization method incorporates curvature estimates into its solution steps, and provides a good approximation to the Newton method for large-scale optimization problems. GGN has been found particularly interesting for practical training of deep neural networks, not only for its impressive convergence speed, but also for its close relation with neural tangent kernel regression, which is central to recent studies that aim to understand the optimization and generalization properties of neural networks. …

arxiv cs.lg gauss math.oc networks neural networks type