Less is More for Improving Automatic Evaluation of Factual Consistency

arXiv:2404.06579v1 Announce Type: cross
Abstract: Assessing the factual consistency of automatically generated texts in relation to source context is crucial for developing reliable natural language generation applications. Recent literature proposes AlignScore which uses a unified alignment model to evaluate factual consistency and substantially outperforms previous methods across many benchmark tasks. In this paper, we take a closer look of datasets used in AlignScore and uncover an unexpected finding: utilizing a smaller number of data points can actually improve performance. We …

abstract alignment applications arxiv benchmark context cs.ai cs.cl cs.lg evaluation generated improving language language generation literature natural natural language natural language generation tasks type