Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed

arXiv:2406.04443v1 Announce Type: new
Abstract: Methods with adaptive stepsizes, such as AdaGrad and Adam, are essential for training modern Deep Learning models, especially Large Language Models. Typically, the noise in the stochastic gradients is heavy-tailed for the later ones. Gradient clipping provably helps to achieve good high-probability convergence for such noises. However, despite the similarity between AdaGrad/Adam and Clip-SGD, the high-probability convergence of AdaGrad/Adam has not been studied in this case. In this work, we prove that AdaGrad (and its …

abstract adam arxiv convergence cs.lg deep learning good gradient language language models large language large language models later math.oc modern noise ones probability stochastic training type