A Careful Examination of Large Language Model Performance on Grade School Arithmetic

arXiv:2405.00332v1 Announce Type: cross
Abstract: Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring …

abstract arxiv benchmark benchmarks cs.ai cs.cl cs.lg data dataset however language language model language models large language large language model large language models leaks llms mathematical reasoning performance questions reasoning school success training training data true type