Sharpness-Aware Minimization and the Edge of Stability

arXiv:2309.12488v5 Announce Type: replace
Abstract: Recent experiments have shown that, often, when training a neural network with gradient descent (GD) with a step size $\eta$, the operator norm of the Hessian of the loss grows until it approximately reaches $2/\eta$, after which it fluctuates around this value. The quantity $2/\eta$ has been called the "edge of stability" based on consideration of a local quadratic approximation of the loss. We perform a similar calculation to arrive at an "edge of stability" …

abstract arxiv cs.lg cs.ne edge gradient loss network neural network norm stability stat.ml the edge training type value