[R] Efficient Streaming Language Models with Attention Sinks - Meta AI 2023 - StreamingLLM enables Llama-2, Falcon and Pythia to have an infinite context length without any fine-tuning! Allows streaming use of LLMs!

Paper: [https://arxiv.org/abs/2309.17453](https://arxiv.org/abs/2309.17453)

Github: [https://github.com/mit-han-lab/streaming-llm](https://github.com/mit-han-lab/streaming-llm)

Abstract:

>Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails …

abstract applications attention caching challenges decoding dialogue interactions language language models large language large language models llms machinelearning major memory popular stage streaming tokens training value