Evaluating Retrieval Quality in Retrieval-Augmented Generation

arXiv:2404.13781v1 Announce Type: new
Abstract: Evaluating retrieval-augmented generation (RAG) presents challenges, particularly for retrieval models within these systems. Traditional end-to-end evaluation methods are computationally expensive. Furthermore, evaluation of the retrieval model's performance based on query-document relevance labels shows a small correlation with the RAG system's downstream performance. We propose a novel evaluation approach, eRAG, where each document in the retrieval list is individually utilized by the large language model within the RAG system. The output generated for each document is …

abstract arxiv challenges correlation cs.cl cs.ir document evaluation labels novel performance quality query rag retrieval retrieval-augmented shows small s performance systems type