Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

arXiv:2403.05293v1 Announce Type: new
Abstract: In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size $\gamma$ and momentum parameter $\beta$ that allows us to identify an intrinsic quantity $\lambda = \frac{ \gamma }{ (1 - \beta)^2 }$ which uniquely defines the optimisation path and provides a simple acceleration rule. When training a $2$-layer diagonal linear network in an overparametrised …

abstract analysis arxiv beta continuous cs.lg gradient identify intrinsic linear math.oc networks optimisation stat.ml training trajectory type work