Long-form factuality in large language models

arXiv:2403.18802v1 Announce Type: cross
Abstract: Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. To benchmark a model's long-form factuality in open domains, we first use GPT-4 to generate LongFact, a prompt set comprising thousands of questions spanning 38 topics. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE). SAFE utilizes an LLM to …

abstract arxiv benchmark cs.ai cs.cl cs.lg domains errors form generate gpt gpt-4 language language models large language large language models llms prompt prompts questions set topics type