Llemma: An Open Language Model For Mathematics

ArXiv | Models | Data | Code | Blog | Sample Explorer
Today we release Llemma: 7 billion and 34 billion parameter language models for mathematics. The Llemma models were initialized with Code Llama weights, then trained on the Proof-Pile II, a 55 billion token dataset of mathematical and scientific documents. The resulting models show improved mathematical capabilities, and can be adapted to various tasks through prompting or additional fine-tuning.

arxiv billion blog code code llama data dataset documents language language model language models llama mathematics release token