CogBench: a large language model walks into a psychology lab

arXiv:2402.18225v1 Announce Type: cross
Abstract: Large language models (LLMs) have significantly advanced the field of artificial intelligence. Yet, evaluating them comprehensively remains challenging. We argue that this is partly due to the predominant focus on performance metrics in most benchmarks. This paper introduces CogBench, a benchmark that includes ten behavioral metrics derived from seven cognitive psychology experiments. This novel approach offers a toolkit for phenotyping LLMs' behavior. We apply CogBench to 35 LLMs, yielding a rich and diverse dataset. We …

abstract advanced artificial artificial intelligence arxiv benchmark benchmarks cs.ai cs.cl cs.lg focus intelligence lab language language model language models large language large language model large language models llms metrics paper performance psychology them type