FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction

arXiv:2403.02270v1 Announce Type: new
Abstract: Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as a substantial number of automatically-generated summaries exhibit factual inconsistencies, such as hallucinations. In response to this issue, various approaches for the evaluation of consistency for summarization have emerged. Yet, these newly-introduced metrics face several limitations, including lack of interpretability, focus on short document summaries (e.g., news articles), and computational impracticality, especially …

abstract arxiv challenge claim cs.cl evaluation extraction generated hallucinations inference issue language language models large language large language models llms natural natural language performance summarization text text summarization type