Robust Implicit Regularization via Weight Normalization

arXiv:2305.05448v3 Announce Type: replace
Abstract: Overparameterized models may have many interpolating solutions; implicit regularization refers to the hidden preference of a particular optimization method towards a certain interpolating solution among the many. A by now established line of work has shown that (stochastic) gradient descent tends to have an implicit bias towards low rank and/or sparse solutions when used to train deep linear networks, explaining to some extent why overparameterized neural network models trained by gradient descent tend to have …

abstract arxiv bias cs.lg gradient hidden line low math.oc normalization optimization regularization robust solution solutions stochastic type via work