Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models

arXiv:2402.15162v1 Announce Type: cross
Abstract: Abstractive summarization models often generate factually inconsistent content particularly when the parametric knowledge of the model conflicts with the knowledge in the input document. In this paper, we analyze the robustness of fine-tuning based summarization models to the knowledge conflict, which we call factual adaptiveness. We utilize pre-trained language models to construct evaluation sets and find that factual adaptiveness is not strongly correlated with factual consistency on original datasets. Furthermore, we introduce a controllable counterfactual …

abstract analyze arxiv call conflict cs.ai cs.cl cs.lg document fine-tuning generate knowledge paper parametric robustness summarization type