Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning

arXiv:2404.01714v1 Announce Type: new
Abstract: Training deep neural networks is a challenging task. In order to speed up training and enhance the performance of deep neural networks, we rectify the vanilla conjugate gradient as conjugate-gradient-like and incorporate it into the generic Adam, and thus propose a new optimization algorithm named CG-like-Adam for deep learning. Specifically, both the first-order and the second-order moment estimation of generic Adam are replaced by the conjugate-gradient-like. Convergence analysis handles the cases where the exponential moving …

abstract adam algorithm arxiv cs.ai cs.cv cs.lg deep learning gradient math.oc networks neural networks optimization performance speed training type