Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias

Recent work in deep learning has shown strong empirical and theoretical evidence of an implicit low-rank bias: weight matrices in deep networks tend to be approximately low-rank and removing relatively small singular values during training or from available trained models may significantly reduce model size while maintaining or even improving model performance. However, the majority of the theoretical investigations around low-rank bias in neural networks deal with oversimplified deep linear networks. In this work, we consider general networks with nonlinear …

bias class cs.lg cs.na deep learning evidence low math.na networks reduce singular small stat.ml training values work