FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizers by Exploiting Strong Convexity. (arXiv:2104.13790v3 [cs.LG] UPDATED)

AdaBelief, one of the current best optimizers, demonstrates superior
generalization ability compared to the popular Adam algorithm by viewing the
exponential moving average of observed gradients. AdaBelief is theoretically
appealing in that it has a data-dependent $O(\sqrt{T})$ regret bound when
objective functions are convex, where $T$ is a time horizon. It remains however
an open problem whether the convergence rate can be further improved without
sacrificing its generalization ability. %on how to exploit strong convexity to
further improve the convergence …

arxiv belief convergence rate