Better Parameter-free Stochastic Optimization with ODE Updates for Coin-Betting. (arXiv:2006.07507v3 [cs.LG] UPDATED)

Web: http://arxiv.org/abs/2006.07507

Parameter-free stochastic gradient descent (PFSGD) algorithms do not require
setting learning rates while achieving optimal theoretical performance. In
practical applications, however, there remains an empirical gap between tuned
stochastic gradient descent (SGD) and PFSGD. In this paper, we close the
empirical gap with a new parameter-free algorithm based on continuous-time
Coin-Betting on truncated models. The new update is derived through the
solution of an Ordinary Differential Equation (ODE) and solved in a closed
form. We show empirically that this new …

arxiv free optimization stochastic