Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation

arXiv:2311.14120v2 Announce Type: replace
Abstract: We investigate the stationary (late-time) training regime of single- and two-layer linear underparameterized neural networks within the continuum limit of stochastic gradient descent (SGD) for synthetic Gaussian data. In the case of a single-layer network in the weakly underparameterized regime, the spectrum of the noise covariance matrix deviates notably from the Hessian, which can be attributed to the broken detailed balance of SGD dynamics. The weight fluctuations are in this case generally anisotropic, but are …

abstract arxiv case cond-mat.dis-nn cond-mat.stat-mech cs.lg data derivation gradient layer linear network networks neural networks stochastic synthetic training type variance