MR-GSM8K: A Meta-Reasoning Revolution in Large Language Model Evaluation

In this work, we introduce a novel evaluation paradigm for Large Language Models, one that challenges them to engage in meta-reasoning. This approach addresses critical shortcomings in existing math problem-solving benchmarks, traditionally used to evaluate the cognitive capabilities of agents. Our paradigm shifts the focus from result-oriented assessments, which often overlook the reasoning process, to a more holistic evaluation that effectively differentiates the cognitive capabilities among models. For example, in our benchmark, GPT-4 demonstrates a performance five times better than …

agents benchmarks capabilities challenges cognitive cs.cl evaluation focus language language model language models large language large language model large language models math meta novel paradigm problem-solving reasoning them work