No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

arXiv:2402.18096v1 Announce Type: new
Abstract: Key-Value (KV) Caching has become an essential technique for accelerating the inference speed and throughput of generative Large Language Models~(LLMs). However, the memory footprint of the KV cache poses a critical bottleneck in LLM deployment as the cache size grows with batch size and sequence length, often surpassing even the size of the model itself. Although recent methods were proposed to select and evict unimportant KV pairs from the cache to reduce memory consumption, the …

abstract arxiv become cache caching compression cs.ai cs.lg deployment generative importance inference key language language models large language large language models llm llms memory mixed precision quantization speed token type value via