KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization

arXiv:2405.03917v1 Announce Type: new
Abstract: Efficient deployment of Large Language Models (LLMs) requires batching multiple requests together to improve throughput. As the batch size, context length, or model size increases, the size of the key and value (KV) cache can quickly become the main contributor to GPU memory usage and the bottleneck of inference latency. Quantization has emerged as an effective technique for KV cache compression, but existing methods still fail at very low bit widths. We observe that distinct …

abstract arxiv batching become cache context context length contributor cs.lg deployment inference key language language model language models large language large language model large language models llms multiple per quantization the key together type value