ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models

arXiv:2403.05266v1 Announce Type: cross
Abstract: Large language models (LLMs) have achieved unprecedented performance in various applications, yet their evaluation remains a critical issue. Existing hallucination benchmarks are either static or lack adjustable complexity for thorough analysis. We contend that utilizing existing relational databases is a promising approach for constructing benchmarks due to their accurate knowledge description via functional dependencies. We propose ERBench to automatically convert any relational database into a benchmark based on the entity-relationship (ER) model. Our key idea …

abstract analysis applications arxiv benchmark benchmarks complexity cs.ai cs.cl cs.lg databases evaluation hallucination issue language language models large language large language models llms performance relational relational databases relationship type