On the fast convergence of minibatch heavy ball momentum. (arXiv:2206.07553v1 [cs.LG])

Web: http://arxiv.org/abs/2206.07553

Simple stochastic momentum methods are widely used in machine learning
optimization, but their good practical performance is at odds with an absence
of theoretical guarantees of acceleration in the literature. In this work, we
aim to close the gap between theory and practice by showing that stochastic
heavy ball momentum, which can be interpreted as a randomized Kaczmarz
algorithm with momentum, retains the fast linear rate of (deterministic) heavy
ball momentum on quadratic optimization problems, at least when minibatching
with …

arxiv convergence lg on