Evaluating the Deductive Competence of Large Language Models

arXiv:2309.05452v2 Announce Type: replace
Abstract: The development of highly fluent large language models (LLMs) has prompted increased interest in assessing their reasoning and problem-solving capabilities. We investigate whether several LLMs can solve a classic type of deductive reasoning problem from the cognitive science literature. The tested LLMs have limited abilities to solve these problems in their conventional form. We performed follow up experiments to investigate if changes to the presentation format and content improve model performance. We do find performance …

abstract arxiv capabilities cognitive cognitive science cs.cl development language language models large language large language models literature llms problem-solving reasoning science solve type