NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding

arXiv:2404.13627v1 Announce Type: new
Abstract: Large Language Models (LLMs) have sparked substantial interest and debate concerning their potential emergence of Theory of Mind (ToM) ability. Theory of mind evaluations currently focuses on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations, which lacks evaluation of machine ToM ability in real-world human interaction scenarios. This poses a pressing demand to develop new real-world scenario benchmarks. We introduce NegotiationToM, a new benchmark designed to stress-test machine ToM …

abstract arxiv benchmark correlations cs.ai cs.cl data emergence game generated language language models large language large language models llms machine mind negotiation stress testing theory theory of mind tom type