Second-order Information Promotes Mini-Batch Robustness in Variance-Reduced Gradients

arXiv:2404.14758v1 Announce Type: cross
Abstract: We show that, for finite-sum minimization problems, incorporating partial second-order information of the objective function can dramatically improve the robustness to mini-batch size of variance-reduced stochastic gradient methods, making them more scalable while retaining their benefits over traditional Newton-type approaches. We demonstrate this phenomenon on a prototypical stochastic second-order algorithm, called Mini-Batch Stochastic Variance-Reduced Newton ($\texttt{Mb-SVRN}$), which combines variance-reduced gradient estimates with access to an approximate Hessian oracle. In particular, we show that when the …

abstract arxiv benefits cs.lg function gradient information making math.oc robustness scalable show stat.ml stochastic sum them type variance