FOSI: Hybrid First and Second Order Optimization

arXiv:2302.08484v4 Announce Type: replace
Abstract: Popular machine learning approaches forgo second-order information due to the difficulty of computing curvature in high dimensions. We present FOSI, a novel meta-algorithm that improves the performance of any base first-order optimizer by efficiently incorporating second-order information during the optimization process. In each iteration, FOSI implicitly splits the function into two quadratic functions defined on orthogonal subspaces, then uses a second-order method to minimize the first, and the base optimizer to minimize the other. We …

abstract algorithm arxiv computing cs.lg dimensions hybrid information iteration machine machine learning meta novel optimization performance popular process type