Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad

arXiv:2403.02648v1 Announce Type: new
Abstract: Adaptive methods are extremely popular in machine learning as they make learning rate tuning less expensive. This paper introduces a novel optimization algorithm named KATE, which presents a scale-invariant adaptation of the well-known AdaGrad algorithm. We prove the scale-invariance of KATE for the case of Generalized Linear Models. Moreover, for general smooth non-convex problems, we establish a convergence rate of $O \left(\frac{\log T}{\sqrt{T}} \right)$ for KATE, matching the best-known ones for AdaGrad and Adam. We …

abstract algorithm arxiv case cs.ai cs.lg kate machine machine learning math.oc novel optimization paper popular prove rate scale square type