Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape. (arXiv:2201.08025v1 [cs.LG])

In this paper, we study the sharpness of a deep learning (DL) loss landscape
around local minima in order to reveal systematic mechanisms underlying the
generalization abilities of DL models. Our analysis is performed across varying
network and optimizer hyper-parameters, and involves a rich family of different
sharpness measures. We compare these measures and show that the low-pass
filter-based measure exhibits the highest correlation with the generalization
abilities of DL models, has high robustness to both data and label noise, …

arxiv deep learning learning optimization