ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition

arXiv:2402.15220v1 Announce Type: new
Abstract: Self-attention is an essential component of large language models(LLMs) but a significant source of inference latency for long sequences. In multi-tenant LLMs serving scenarios, the compute and memory operation cost of self-attention can be optimized by using the probability that multiple LLM requests have shared system prompts in prefixes. In this paper, we introduce ChunkAttention, a prefix-aware self-attention module that can detect matching prompt prefixes across multiple requests and share their key/value tensors in memory …

abstract arxiv attention cache compute cost cs.cl cs.lg inference inference latency language language models large language large language models latency llm llms memory multiple probability self-attention type