How You Start Matters for Generalization. (arXiv:2206.08558v1 [cs.LG])

Web: http://arxiv.org/abs/2206.08558

Characterizing the remarkable generalization properties of over-parameterized
neural networks remains an open problem. In this paper, we promote a shift of
focus towards initialization rather than neural architecture or (stochastic)
gradient descent to explain this implicit regularization. Through a Fourier
lens, we derive a general result for the spectral bias of neural networks and
show that the generalization of neural networks is heavily tied to their
initialization. Further, we empirically solidify the developed theoretical
insights using practical, deep networks. Finally, …

arxiv lg