On the Stability of Gradient Descent for Large Learning Rate

arXiv:2402.13108v1 Announce Type: new
Abstract: There currently is a significant interest in understanding the Edge of Stability (EoS) phenomenon, which has been observed in neural networks training, characterized by a non-monotonic decrease of the loss function over epochs, while the sharpness of the loss (spectral norm of the Hessian) progressively approaches and stabilizes around 2/(learning rate). Reasons for the existence of EoS when training using gradient descent have recently been proposed -- a lack of flat minima near the gradient …

abstract arxiv cs.lg edge function gradient loss networks neural networks norm rate stability the edge training type understanding