On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions

The Adaptive Momentum Estimation (Adam) algorithm is highly effective in training various deep learning tasks. Despite this, there's limited theoretical understanding for Adam, especially when focusing on its vanilla form in non-convex smooth scenarios with potential unbounded gradients and affine variance noise. In this paper, we study vanilla Adam under these challenging conditions. We introduce a comprehensive noise model which governs affine variance noise, bounded noise and sub-Gaussian noise. We show that Adam can find a stationary point with a …

adam algorithm assumptions convergence cs.lg deep learning form math.oc noise optimization paper stat.ml stochastic study tasks training understanding variance