Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training

Given the massive cost of language model pre-training, a non-trivial
improvement of the optimization algorithm would lead to a material
reduction on the time and cost of training. Adam and its variants have been
state-of-the-art for years, and more sophisticated second-order
(Hessian-based) optimizers often incur too much per-step overhead.

adam algorithm art cost improvement language language model massive material optimization per pre-training research scalable state stochastic training variants