Learning time-scales in two-layers neural networks

arXiv:2303.00055v3 Announce Type: replace
Abstract: Gradient-based learning in multi-layer neural networks displays a number of striking features. In particular, the decrease rate of empirical risk is non-monotone even after averaging over large batches. Long plateaus in which one observes barely any progress alternate with intervals of rapid decrease. These successive phases of learning often take place on very different time scales. Finally, models learnt in an early phase are typically `simpler' or `easier to learn' although in a way that …

abstract arxiv cs.lg features gradient layer math.oc networks neural networks progress rate risk stat.ml type