Sequence can Secretly Tell You What to Discard

arXiv:2404.15949v1 Announce Type: new
Abstract: Large Language Models (LLMs), despite their impressive performance on a wide range of tasks, require significant GPU memory and consume substantial computational resources. In addition to model weights, the memory occupied by KV cache increases linearly with sequence length, becoming a main bottleneck for inference. In this paper, we introduce a novel approach for optimizing the KV cache which significantly reduces its memory footprint. Through a comprehensive investigation, we find that on LLaMA2 series models, …

abstract arxiv cache computational cs.ai cs.cl cs.lg gpu inference language language models large language large language models llms memory performance resources tasks type