FABLES: Evaluating faithfulness and content selection in book-length summarization

arXiv:2404.01261v1 Announce Type: new
Abstract: While long-context large language models (LLMs) can technically summarize book-length documents (>100K tokens), the length and complexity of the documents have so far prohibited evaluations of input-dependent aspects like faithfulness. In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books. Our study mitigates the issue of data contamination by focusing on summaries of books published in 2023 or 2024, and we hire annotators who …

abstract arxiv book complexity context cs.ai cs.cl documents evaluation human language language models large language large language models llms paper scale summarization tokens type