Black Box Lie Group Preconditioners for SGD. (arXiv:2211.04422v1 [stat.ML])

A matrix free and a low rank approximation preconditioner are proposed to
accelerate the convergence of stochastic gradient descent (SGD) by exploiting
curvature information sampled from Hessian-vector products or finite
differences of parameters and gradients similar to the BFGS algorithm. Both
preconditioners are fitted with an online updating manner minimizing a
criterion that is free of line search and robust to stochastic gradient noise,
and further constrained to be on certain connected Lie groups to preserve their
corresponding symmetry or …

arxiv black box