FlashBack:Efficient Retrieval-Augmented Language Modeling for Long Context Inference

arXiv:2405.04065v1 Announce Type: new
Abstract: Retrieval-Augmented Language Modeling (RALM) by integrating large language models (LLM) with relevant documents from an external corpus is a proven method for enabling the LLM to generate information beyond the scope of its pre-training corpus. Previous work using utilizing retrieved content by simply prepending retrieved contents to the input poses a high runtime issue, which degrades the inference efficiency of the LLMs because they fail to use the Key-Value (KV) cache efficiently. In this paper, …

abstract arxiv beyond contents context cs.cl documents enabling generate inference information language language models large language large language models llm modeling pre-training retrieval retrieval-augmented training type work