On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond

arXiv:2403.15146v1 Announce Type: new
Abstract: This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates. We demonstrate that Adam achieves a faster convergence compared to SGDM under the condition of non-uniformly bounded smoothness. Our findings reveal that: (1) in deterministic environments, Adam can attain the known lower bound for the convergence rate of deterministic first-order optimizers, whereas the convergence rate of Gradient Descent with Momentum (GDM) has higher order …

abstract adam arxiv beyond convergence cs.lg faster gradient math.oc paper stochastic terms type uniform