RORA: Robust Free-Text Rationale Evaluation

arXiv:2402.18678v1 Announce Type: new
Abstract: Free-text rationales play a pivotal role in explainable NLP, bridging the knowledge and reasoning gaps behind a model's decision-making. However, due to the diversity of potential reasoning paths and a corresponding lack of definitive ground truth, their evaluation remains a challenge. Existing evaluation metrics rely on the degree to which a rationale supports a target label, but we find these fall short in evaluating rationales that inadvertently leak the labels. To address this problem, we …

abstract arxiv challenge cs.cl decision diversity evaluation evaluation metrics free knowledge making metrics nlp pivotal reasoning robust role text truth type