High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance

arXiv:2304.00707v2 Announce Type: replace-cross
Abstract: We derive high-dimensional scaling limits and fluctuations for the online least-squares Stochastic Gradient Descent (SGD) algorithm by taking the properties of the data generating model explicitly into consideration. Our approach treats the SGD iterates as an interacting particle system, where the expected interaction is characterized by the covariance structure of the input. Assuming smoothness conditions on moments of order up to eight orders, and without explicitly assuming Gaussianity, we establish the high-dimensional scaling limits and …

abstract algorithm arxiv covariance data gradient least math.oc math.pr math.st particle scaling squares stat.ml stat.th stochastic type