A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness

arXiv:2403.15244v1 Announce Type: new
Abstract: Classical convergence analyses for optimization algorithms rely on the widely-adopted uniform smoothness assumption. However, recent experimental studies have demonstrated that many machine learning problems exhibit non-uniform smoothness, meaning the smoothness factor is a function of the model parameter instead of a universal constant. In particular, it has been observed that the smoothness grows with respect to the gradient norm along the training trajectory. Motivated by this phenomenon, the recently introduced $(L_0, L_1)$-smoothness is a more …

abstract algorithms arxiv convergence cs.lg experimental function however machine machine learning math.oc meaning optimization stochastic studies type uniform universal