SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix

arXiv:2403.02833v1 Announce Type: new
Abstract: This paper introduces a new stochastic optimization method based on the regularized Fisher information matrix (FIM), named SOFIM, which can efficiently utilize the FIM to approximate the Hessian matrix for finding Newton's gradient update in large-scale stochastic optimization of machine learning models. It can be viewed as a variant of natural gradient descent (NGD), where the challenge of storing and calculating the full FIM is addressed through making use of the regularized FIM and directly …

abstract arxiv cs.lg cs.ne fisher gradient information machine machine learning machine learning models matrix optimization paper scale stochastic type update