Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives

arXiv:2402.11051v1 Announce Type: new
Abstract: Existing datasets for narrative understanding often fail to represent the complexity and uncertainty of relationships in real-life social scenarios. To address this gap, we introduce a new benchmark, Conan, designed for extracting and analysing intricate character relation graphs from detective narratives. Specifically, we designed hierarchical relationship categories and manually extracted and annotated role-oriented relationships from the perspectives of various characters, incorporating both public relationships known to most characters and secret ones known to only a …

abstract arxiv benchmark complexity cs.ai cs.cl datasets gap graphs language language models large language large language models life narrative relationships social type uncertainty understanding