Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference

arXiv:2402.09398v1 Announce Type: new
Abstract: Many computational factors limit broader deployment of large language models. In this paper, we focus on a memory bottleneck imposed by the key-value (KV) cache, a computational shortcut that requires storing previous KV pairs during decoding. While existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs to dramatically reduce the memory footprint of the cache, they can have limited success in tasks that require recollecting …

abstract arxiv cache compression computational cs.ai cs.lg decoding deployment focus inference key language language models large language large language models llm memory paper shortcut the key type value