QLABGrad: a Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learning

arXiv:2302.00252v2 Announce Type: replace
Abstract: The learning rate is a critical hyperparameter for deep learning tasks since it determines the extent to which the model parameters are updated during the learning course. However, the choice of learning rates typically depends on empirical judgment, which may not result in satisfactory outcomes without intensive try-and-error experiments. In this study, we propose a novel learning rate adaptation scheme called QLABGrad. Without any user-specified hyperparameter, QLABGrad automatically determines the learning rate by optimizing the …

abstract arxiv convergence course cs.lg deep learning free however hyperparameter judgment math.oc parameters rate tasks type