Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks. (arXiv:2201.04738v1 [stat.ML])

We study the dynamics of a neural network in function space when optimizing
the mean squared error via gradient flow. We show that in the
underparameterized regime the network learns eigenfunctions of an integral
operator $T_{K^\infty}$ determined by the Neural Tangent Kernel (NTK) at rates
corresponding to their eigenvalues. For example, for uniformly distributed data
on the sphere $S^{d - 1}$ and rotation invariant weight distributions, the
eigenfunctions of $T_{K^\infty}$ are the spherical harmonics. Our results can
be understood as …

arxiv bias gradient ml networks neural networks optimization