AttentionStore: Cost-effective Attention Reuse across Multi-turn Conversations in Large Language Model Serving

arXiv:2403.19708v1 Announce Type: cross
Abstract: Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines for executing multi-turn conversations are inefficient due to the need to repeatedly compute the key-value (KV) caches of historical tokens, incurring high serving costs. To address the problem, this paper proposes AttentionStore, a new attention mechanism that enables the reuse of KV caches (i.e., attention reuse) across multi-turn conversations, significantly reducing the repetitive computation overheads. …

abstract arxiv attention compute conversations cost cs.cl cs.lg feature however humans key language language model language models large language large language model large language models llm llms the key through tokens type value