SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

arXiv:2404.04793v1 Announce Type: new
Abstract: Optimizing the Key-Value (KV) cache of the Large Language Model (LLM) has been considered critical to saving the cost of inference. Most of the existing KV-cache compression algorithms attempted to sparsify the sequence of tokens by taking advantage of the different importance of tokens. In this work, we found that by identifying the importance of attention layers, we could optimize the KV-cache jointly from two dimensions. Based on our observations regarding layer-wise importance in inference, …

arxiv budget cache cs.cl cs.lg inference layer llm management type via wise