SnapKV: LLM Knows What You are Looking for Before Generation

arXiv:2404.14469v1 Announce Type: new
Abstract: Large Language Models (LLMs) have made remarkable progress in processing extensive contexts, with the Key-Value (KV) cache playing a vital role in enhancing their performance. However, the growth of the KV cache in response to increasing input length poses challenges to memory and time efficiency. To address this problem, this paper introduces SnapKV, an innovative and fine-tuning-free approach that efficiently minimizes KV cache size while still delivering comparable performance in real-world applications.
We discover that …

abstract arxiv cache challenges cs.ai cs.cl efficiency growth however key language language models large language large language models llm llms memory performance playing processing progress role the key type value vital