SGD and Weight Decay Provably Induce a Low-Rank Bias in Neural Networks. (arXiv:2206.05794v2 [cs.LG] UPDATED)

We analyze deep ReLU neural networks trained with mini-batch Stochastic
Gradient Descent (SGD) and weight decay. We show, both theoretically and
empirically, that when training a neural network using SGD with weight decay
and small batch size, the resulting weight matrices tend to be of small rank.
Our analysis relies on a minimal set of assumptions; the neural networks may be
arbitrarily wide or deep and may include residual connections, as well as
convolutional layers. The same analysis implies the …

arxiv bias low networks neural networks