UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models

arXiv:2402.14690v1 Announce Type: new
Abstract: Large language models (LLMs) may generate text that lacks consistency with human knowledge, leading to factual inaccuracies or \textit{hallucination}. Existing research for evaluating the factuality of LLMs involves extracting fact claims using an LLM and verifying them against a predefined fact source. However, these evaluation metrics are task-specific, and not scalable, and the substitutability of fact sources in different tasks is under-explored. To address these challenges, we categorize four available fact sources: human-written evidence, reference …

arxiv cs.cl framework language language models large language large language models type ufo