GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations

arXiv:2402.12348v1 Announce Type: cross
Abstract: As Large Language Models (LLMs) are integrated into critical real-world applications, their strategic and logical reasoning abilities are increasingly crucial. This paper evaluates LLMs' reasoning abilities in competitive environments through game-theoretic tasks, e.g., board and card games that require pure logic and strategic reasoning to compete with opponents. We first propose GTBench, a language-driven environment composing 10 widely-recognized tasks, across a comprehensive game taxonomy: complete versus incomplete information, dynamic versus static, and probabilistic versus deterministic …

abstract applications arxiv board card cs.ai cs.cl cs.lg environments game games language language models large language large language models limitations llms logic paper reasoning tasks through type via world