Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

arXiv:2404.07143v1 Announce Type: cross
Abstract: This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling …

abstract arxiv attention computation context cs.ai cs.cl cs.lg cs.ne inputs key language language models large language large language models llms memory scale transformer transformers type work