Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts

arXiv:2310.05898v5 Announce Type: replace
Abstract: Lion (Evolved Sign Momentum), a new optimizer discovered through program search, has shown promising results in training large AI models. It performs comparably or favorably to AdamW but with greater memory efficiency. As we can expect from the results of a random search program, Lion incorporates elements from several existing algorithms, including signed momentum, decoupled weight decay, Polak, and Nesterov momentum, but does not fit into any existing category of theoretically grounded optimizers. Thus, even …

abstract ai models arxiv cs.ai cs.lg efficiency expect math.oc memory optimization random results search stat.ap stat.ml through training type