Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level

arXiv:2404.05337v1 Announce Type: new
Abstract: Prominent large language models have exhibited human-level performance in many domains, even enabling the derived agents to simulate human and social interactions. While practical works have substantiated the practicability of grounding language agents in sandbox simulation or embodied simulators, current social intelligence benchmarks either stay at the language level or use subjective metrics. In pursuit of a more realistic and objective evaluation, we introduce the Social Tasks in Sandbox Simulation (STSS) benchmark, which assesses language …

abstract agents arxiv benchmarking benchmarks cs.ai cs.cl current domains embodied enabling human intelligence interactions language language models large language large language models performance practical simulation social type