Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training

Wrote up a blog post on the new second-order optimizer Sophia, which is showing encouraging results on LLM pretraining.

This paper has some good use of advanced optimization theory, the resources for which I have included in my blog.

Blog - [https://shreyansh26.github.io/post/2023-05-28\_sophia\_scalable\_second\_order\_optimizer\_llms/](https://shreyansh26.github.io/post/2023-05-28_sophia_scalable_second_order_optimizer_llms/)

Annotated Paper - [Sophia Annotated Paper - Github](https://github.com/shreyansh26/Annotated-ML-Papers/blob/main/ML%20Theory/Sophia%20-%20A%20Scalable%20Stochastic%20Second-order%20Optimizer%20for%20Language%20Model%20Pretraining.pdf)

advanced artificial blog good language language model llm optimization paper pre-training resources scalable stochastic theory training