The Implicit Regularization of Momentum Gradient Descent with Early Stopping. (arXiv:2201.05405v1 [cs.LG])

The study on the implicit regularization induced by gradient-based
optimization is a longstanding pursuit. In the present paper, we characterize
the implicit regularization of momentum gradient descent (MGD) with early
stopping by comparing with the explicit $\ell_2$-regularization (ridge). In
details, we study MGD in the continuous-time view, so-called momentum gradient
flow (MGF), and show that its tendency is closer to ridge than the gradient
descent (GD) [Ali et al., 2019] for least squares regression. Moreover, we
prove that, under the …

arxiv gradient