Making Retrieval-Augmented Language Models Robust to Irrelevant Context

arXiv:2310.01558v2 Announce Type: replace
Abstract: Retrieval-augmented language models (RALMs) hold promise to produce language understanding systems that are are factual, efficient, and up-to-date. An important desideratum of RALMs, is that retrieved information helps model performance when it is relevant, and does not harm performance when it is not. This is particularly important in multi-hop reasoning scenarios, where misuse of irrelevant evidence can lead to cascading errors. However, recent work has shown that retrieval augmentation can sometimes have a negative effect …

abstract arxiv context cs.ai cs.cl harm information language language models language understanding making performance retrieval retrieval-augmented robust systems type understanding