QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization. (arXiv:2112.08542v2 [cs.CL] UPDATED)

Factual consistency is an essential quality of text summarization models in
practical settings. Existing work in evaluating this dimension can be broadly
categorized into two lines of research, entailment-based and question answering
(QA)-based metrics, and different experimental setups often lead to contrasting
conclusions as to which paradigm performs the best. In this work, we conduct an
extensive comparison of entailment and QA-based metrics, demonstrating that
carefully choosing the components of a QA-based metric, especially question
generation and answerability classification, is …

arxiv evaluation qa summarization