On the Trajectories of SGD Without Replacement

arXiv:2312.16143v2 Announce Type: replace
Abstract: This article examines the implicit regularization effect of Stochastic Gradient Descent (SGD). We consider the case of SGD without replacement, the variant typically used to optimize large-scale neural networks. We analyze this algorithm in a more realistic regime than typically considered in theoretical works on SGD, as, e.g., we allow the product of the learning rate and Hessian to be $O(1)$ and we do not specify any model architecture, learning task, or loss (objective) function. …

abstract algorithm analyze article arxiv case cs.lg gradient math.oc networks neural networks regularization replacement scale stat.ml stochastic type