Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation

arXiv:2405.03085v1 Announce Type: new
Abstract: Large Language Models (LLMs) have made significant strides in information acquisition. However, their overreliance on potentially flawed parametric knowledge leads to hallucinations and inaccuracies, particularly when handling long-tail, domain-specific queries. Retrieval Augmented Generation (RAG) addresses this limitation by incorporating external, non-parametric knowledge. Nevertheless, the retrieved long-context documents often contain noisy, irrelevant information alongside vital knowledge, negatively diluting LLMs' attention. Inspired by the supportive role of essential concepts in individuals' reading comprehension, we propose a novel …

abstract acquisition amr arxiv concept context cs.cl distillation domain hallucinations however information knowledge language language models large language large language models leads llms non-parametric parametric queries rag retrieval retrieval augmented generation type