GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM

arXiv:2403.05527v1 Announce Type: new
Abstract: Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to be a memory bound problem, significantly constraining the system throughput. Existing methods rely on dropping unimportant tokens or quantizing all entries uniformly. Such methods, however, often incur high approximation errors to represent the compressed matrices. The autoregressive decoding process further compounds the error of …

arxiv cache compression cs.ai cs.cl cs.lg gear generative inference llm near type