Series of Hessian-Vector Products for Tractable Saddle-Free Newton Optimisation of Neural Networks

arXiv:2310.14901v2 Announce Type: replace
Abstract: Despite their popularity in the field of continuous optimisation, second-order quasi-Newton methods are challenging to apply in machine learning, as the Hessian matrix is intractably large. This computational burden is exacerbated by the need to address non-convexity, for instance by modifying the Hessian's eigenvalues as in Saddle-Free Newton methods. We propose an optimisation algorithm which addresses both of these concerns - to our knowledge, the first efficiently-scalable optimisation algorithm to asymptotically use the exact inverse …

abstract apply arxiv computational continuous cs.lg free instance machine machine learning matrix networks neural networks optimisation products series stat.ml tractable type vector