In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs Miss

arXiv:2402.10790v1 Announce Type: cross
Abstract: This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to $10^4$ elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to …

abstract arxiv benchmark capabilities challenge cs.ai cs.cl cs.lg distributed documents evaluation facts generative haystack llms memory paper processing search transformer transformer models type