HANS, are you clever? Clever Hans Effect Analysis of Neural Systems

arXiv:2309.12481v2 Announce Type: replace
Abstract: Instruction-tuned Large Language Models (It-LLMs) have been exhibiting outstanding abilities to reason around cognitive states, intentions, and reactions of all people involved, letting humans guide and comprehend day-to-day social interactions effectively. In fact, several multiple-choice questions (MCQ) benchmarks have been proposed to construct solid assessments of the models' abilities. However, earlier works are demonstrating the presence of inherent "order bias" in It-LLMs, posing challenges to the appropriate evaluation. In this paper, we investigate It-LLMs' resilience …

abstract analysis arxiv benchmarks cognitive construct cs.ai cs.cl guide hans humans instruction-tuned interactions language language models large language large language models llms multiple people questions reason social solid systems type