BaRDa: A Belief and Reasoning Dataset that Separates Factual Accuracy and Reasoning Ability

arXiv:2312.07527v2 Announce Type: replace
Abstract: While there are numerous benchmarks comparing the performance of modern language models (LMs), end-task evaluations often conflate notions of *factual accuracy* ("truth") and *reasoning ability* ("rationality", or "honesty" in the sense of correctly reporting implications of beliefs). Our goal is a dataset that clearly distinguishes these two notions. Our approach is to leverage and extend a collection of human-annotated *entailment trees*, engineered to express both good and bad chains of reasoning, and using a mixture …

abstract accuracy arxiv belief benchmarks cs.ai cs.cl dataset honesty language language models lms modern performance reasoning reporting sense truth type