High-dimensional limit theorems for SGD: Effective dynamics and critical scaling. (arXiv:2206.04030v1 [stat.ML])

We study the scaling limits of stochastic gradient descent (SGD) with
constant step-size in the high-dimensional regime. We prove limit theorems for
the trajectories of summary statistics (i.e., finite-dimensional functions) of
SGD as the dimension goes to infinity. Our approach allows one to choose the
summary statistics that are tracked, the initialization, and the step-size. It
yields both ballistic (ODE) and diffusive (SDE) limits, with the limit
depending dramatically on the former choices. Interestingly, we find a critical
scaling regime …

arxiv dynamics ml scaling