ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation

arXiv:2405.04818v1 Announce Type: new
Abstract: Evaluating free-text explanations is a multifaceted, subjective, and labor-intensive task. Large language models (LLMs) present an appealing alternative due to their potential for consistency, scalability, and cost-efficiency. In this work, we present ACORN, a new dataset of 3,500 free-text explanations and aspect-wise quality ratings, and use it to gain insights into how LLMs evaluate explanations. We observed that replacing one of the human ratings sometimes maintained, but more often lowered the inter-annotator agreement across different …

arxiv commonsense cs.cl evaluation reasoning type wise