Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey

arXiv:2404.01869v1 Announce Type: new
Abstract: Large language models (LLMs) have recently shown impressive performance on tasks involving reasoning, leading to a lively debate on whether these models possess reasoning capabilities similar to humans. However, despite these successes, the depth of LLMs' reasoning abilities remains uncertain. This uncertainty partly stems from the predominant focus on task performance, measured through shallow accuracy metrics, rather than a thorough investigation of the models' reasoning behavior. This paper seeks to address this gap by providing …

abstract accuracy arxiv behavior beyond capabilities cs.ai cs.cl however humans language language models large language large language models llms performance reasoning survey tasks type uncertain uncertainty