High-dimensional limit theorems for SGD: Effective dynamics and critical scaling. (arXiv:2206.04030v2 [stat.ML] UPDATED)

We study the scaling limits of stochastic gradient descent (SGD) with
constant step-size in the high-dimensional regime. We prove limit theorems for
the trajectories of summary statistics (i.e., finite-dimensional functions) of
SGD as the dimension goes to infinity. Our approach allows one to choose the
summary statistics that are tracked, the initialization, and the step-size. It
yields both ballistic (ODE) and diffusive (SDE) limits, with the limit
depending dramatically on the former choices. We show a critical scaling regime
for …

arxiv dynamics scaling