HesScale: Scalable Computation of Hessian Diagonals. (arXiv:2210.11639v2 [cs.LG] UPDATED)

Second-order optimization uses curvature information about the objective
function, which can help in faster convergence. However, such methods typically
require expensive computation of the Hessian matrix, preventing their usage in
a scalable way. The absence of efficient ways of computation drove the most
widely used methods to focus on first-order approximations that do not capture
the curvature information. In this paper, we develop HesScale, a scalable
approach to approximating the diagonal of the Hessian matrix, to incorporate
second-order information in …

arxiv computation scalable