vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

arXiv:2405.04437v1 Announce Type: new
Abstract: Efficient use of GPU memory is essential for high throughput LLM inference. Prior systems reserved memory for the KV-cache ahead-of-time, resulting in wasted capacity due to internal fragmentation. Inspired by OS-based virtual memory systems, vLLM proposed PagedAttention to enable dynamic memory allocation for KV-cache. This approach eliminates fragmentation, enabling high-throughput LLM serving with larger batch sizes. However, to be able to allocate physical memory dynamically, PagedAttention changes the layout of KV-cache from contiguous virtual memory …

abstract arxiv cache capacity cs.lg cs.os dynamic fragmentation gpu inference llm llms management memory prior systems type virtual