Implicit Regularization via Spectral Neural Networks and Non-linear Matrix Sensing

arXiv:2402.17595v1 Announce Type: new
Abstract: The phenomenon of implicit regularization has attracted interest in recent years as a fundamental aspect of the remarkable generalizing ability of neural networks. In a nutshell, it entails that gradient descent dynamics in many neural nets, even without any explicit regularizer in the loss function, converges to the solution of a regularized learning problem. However, known results attempting to theoretically explain this phenomenon focus overwhelmingly on the setting of linear neural nets, and the simplicity …

abstract arxiv cs.ai cs.lg cs.ne dynamics function gradient linear loss matrix networks neural nets neural networks non-linear regularization sensing stat.ml type via