Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods

arXiv:2312.08531v2 Announce Type: replace
Abstract: In the past several years, the last-iterate convergence of the Stochastic Gradient Descent (SGD) algorithm has triggered people's interest due to its good performance in practice but lack of theoretical understanding. For Lipschitz convex functions, different works have established the optimal $O(\log(1/\delta)\log T/\sqrt{T})$ or $O(\sqrt{\log(1/\delta)/T})$ high-probability convergence rates for the final iterate, where $T$ is the time horizon and $\delta$ is the failure probability. However, to prove these bounds, all the existing works are either …

abstract algorithm arxiv convergence cs.lg delta functions good gradient iterate math.oc people performance practice probability stat.ml stochastic type understanding