On the Maximum Hessian Eigenvalue and Generalization. (arXiv:2206.10654v1 [cs.LG])

The mechanisms by which certain training interventions, such as increasing
learning rates and applying batch normalization, improve the generalization of
deep networks remains a mystery. Prior works have speculated that "flatter"
solutions generalize better than "sharper" solutions to unseen data, motivating
several metrics for measuring flatness (particularly $\lambda_{max}$, the
largest eigenvalue of the Hessian of the loss); and algorithms, such as
Sharpness-Aware Minimization (SAM) [1], that directly optimize for flatness.
Other works question the link between $\lambda_{max}$ and generalization. In …

arxiv eigenvalue lg