Llemma: An Open Language Model For Mathematics

arXiv:2310.10631v3 Announce Type: replace
Abstract: We present Llemma, a large language model for mathematics. We continue pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web data containing mathematics, and mathematical code, yielding Llemma. On the MATH benchmark Llemma outperforms all known open base models, as well as the unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is capable of tool use and formal theorem proving without any further finetuning. We openly release all artifacts, including …

abstract arxiv benchmark code code llama cs.ai cs.cl cs.lo data language language model large language large language model llama math mathematics minerva papers pretraining scientific type web