Prodigy: An Expeditiously Adaptive Parameter-Free Learner

arXiv:2306.06101v4 Announce Type: replace
Abstract: We consider the problem of estimating the learning rate in adaptive methods, such as AdaGrad and Adam. We propose Prodigy, an algorithm that provably estimates the distance to the solution $D$, which is needed to set the learning rate optimally. At its core, Prodigy is a modification of the D-Adaptation method for learning-rate-free learning. It improves upon the convergence rate of D-Adaptation by a factor of $O(\sqrt{\log(D/d_0)})$, where $d_0$ is the initial estimate of $D$. …

abstract adam algorithm arxiv core cs.ai cs.lg free math.oc prodigy rate set solution stat.ml type