Understanding Gradient Descent on Edge of Stability in Deep Learning. (arXiv:2205.09745v1 [cs.LG])

Deep learning experiments in Cohen et al. (2021) using deterministic Gradient
Descent (GD) revealed an {\em Edge of Stability (EoS)} phase when learning rate
(LR) and sharpness (\emph{i.e.}, the largest eigenvalue of Hessian) no longer
behave as in traditional optimization. Sharpness stabilizes around $2/$LR and
loss goes up and down across iterations, yet still with an overall downward
trend. The current paper mathematically analyzes a new mechanism of implicit
regularization in the EoS phase, whereby GD updates due to non-smooth …

arxiv deep learning edge gradient learning understanding