Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

arXiv:2311.09000v3 Announce Type: replace
Abstract: The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence …

arxiv benchmark checkers cs.cl evaluation fine-grained type