Singular-limit analysis of gradient descent with noise injection

arXiv:2404.12293v1 Announce Type: new
Abstract: We study the limiting dynamics of a large class of noisy gradient descent systems in the overparameterized regime. In this regime the set of global minimizers of the loss is large, and when initialized in a neighbourhood of this zero-loss set a noisy gradient descent algorithm slowly evolves along this set. In some cases this slow evolution has been related to better generalisation properties. We characterize this evolution for the broad class of noisy gradient …

abstract algorithm analysis arxiv class cs.lg dynamics global gradient loss math.pr noise set singular study systems type