Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction. (arXiv:2206.07085v1 [cs.LG])

Normalization layers (e.g., Batch Normalization, Layer Normalization) were
introduced to help with optimization difficulties in very deep nets, but they
clearly also help generalization, even in not-so-deep nets. Motivated by the
long-held belief that flatter minima lead to better generalization, this paper
gives mathematical analysis and supporting experiments suggesting that
normalization (together with accompanying weight-decay) encourages GD to reduce
the sharpness of loss surface. Here "sharpness" is carefully defined given that
the loss is scale-invariant, a known consequence of normalization. …

arxiv lg normalization understanding