Views Are My Own, But Also Yours: Benchmarking Theory of Mind using Common Ground

arXiv:2403.02451v1 Announce Type: new
Abstract: Evaluating the theory of mind (ToM) capabilities of language models (LMs) has recently received much attention. However, many existing benchmarks rely on synthetic data which risks misaligning the resulting experiments with human behavior. We introduce the first ToM dataset based on naturally occurring spoken dialogs, Common-ToM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on Common-ToM.

abstract arxiv attention behavior benchmarking benchmarks capabilities cs.cl data dataset human language language models lms mind risks synthetic synthetic data theory theory of mind tom type