FaaF: Facts as a Function for the evaluation of RAG systems

arXiv:2403.03888v1 Announce Type: new
Abstract: Factual recall from a reference source is crucial for evaluating the performance of Retrieval Augmented Generation (RAG) systems, as it directly probes into the quality of both retrieval and generation. However, it still remains a challenge to perform this evaluation reliably and efficiently. Recent work has focused on fact verification via prompting language model (LM) evaluators, however we demonstrate that these methods are unreliable in the presence of incomplete or inaccurate information. We introduce Facts …

abstract arxiv challenge cs.cl evaluation facts function however performance quality rag recall reference retrieval retrieval augmented generation systems type