SmartPlay: A Benchmark for LLMs as Intelligent Agents

arXiv:2310.01557v4 Announce Type: replace
Abstract: Recent large language models (LLMs) have demonstrated great potential toward intelligent agents and next-gen automation, but there currently lacks a systematic benchmark for evaluating LLMs' abilities as agents. We introduce SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs as agents. SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay …

abstract agents arxiv automation benchmark cs.ai cs.lg gen intelligent language language models large language large language models llms methodology next next-gen type