Adaptive Gradient Methods at the Edge of Stability

arXiv:2207.14484v2 Announce Type: replace
Abstract: Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning. In this paper, we shed light on the behavior of these algorithms in the full-batch and sufficiently large batch settings. Specifically, we empirically demonstrate that during full-batch training, the maximum eigenvalue of the preconditioned Hessian typically equilibrates at a certain numerical value -- the stability threshold of a gradient descent algorithm. For Adam with step size $\eta$ and …

abstract adam algorithms arxiv behavior cs.lg deep learning dynamics edge gradient light paper stability the edge training type