NoMIRACL: Knowing When You Don't Know for Robust Multilingual Retrieval-Augmented Generation

arXiv:2312.11361v2 Announce Type: replace
Abstract: Retrieval-augmented generation (RAG) grounds large language model (LLM) output by leveraging external knowledge sources to reduce factual hallucinations. However, prior works lack a comprehensive evaluation of different language families, making it challenging to evaluate LLM robustness against errors in external retrieved knowledge. To overcome this, we establish NoMIRACL, a human-annotated dataset for evaluating LLM robustness in RAG across 18 typologically diverse languages. NoMIRACL includes both a non-relevant and a relevant subset. Queries in the non-relevant …

abstract arxiv cs.cl cs.ir errors evaluation families hallucinations knowledge language language model large language large language model llm making multilingual prior rag reduce retrieval retrieval-augmented robust robustness type