CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understanding

arXiv:2107.09847v2 Announce Type: replace
Abstract: We introduce CogME, a cognition-inspired, multi-dimensional evaluation metric designed for AI models focusing on story understanding. CogME is a framework grounded in human thinking strategies and story elements that involve story understanding. With a specific breakdown of the questions, this approach provides a nuanced assessment revealing not only AI models' particular strengths and weaknesses but also the characteristics of the benchmark dataset. Our case study with the DramaQA dataset demonstrates a refined analysis of the …

abstract ai models arxiv assessment breakdown cognition cs.ai cs.cv evaluation framework human questions story strategies thinking type understanding