On the different regimes of Stochastic Gradient Descent

arXiv:2309.10688v4 Announce Type: replace
Abstract: Modern deep networks are trained with stochastic gradient descent (SGD) whose key hyperparameters are the number of data considered at each step or batch size $B$, and the step size or learning rate $\eta$. For small $B$ and large $\eta$, SGD corresponds to a stochastic evolution of the parameters, whose noise amplitude is governed by the ''temperature'' $T\equiv \eta/B$. Yet this description is observed to break down for sufficiently large batches $B\geq B^*$, or simplifies …

abstract arxiv cond-mat.dis-nn cs.lg data evolution gradient key modern networks rate small stat.ml stochastic type