Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees

arXiv:2305.03938v2 Announce Type: replace-cross
Abstract: In this paper, we present a comprehensive study on the convergence properties of Adam-family methods for nonsmooth optimization, especially in the training of nonsmooth neural networks. We introduce a novel two-timescale framework that adopts a two-timescale updating scheme, and prove its convergence properties under mild assumptions. Our proposed framework encompasses various popular Adam-family methods, providing convergence guarantees for these methods in training nonsmooth neural networks. Furthermore, we develop stochastic subgradient methods that incorporate gradient clipping …

abstract adam arxiv assumptions convergence cs.lg family framework math.oc networks neural networks novel optimization paper prove stat.ml study timescale training type