QAQ: Quality Adaptive Quantization for LLM KV Cache

arXiv:2403.04643v1 Announce Type: new
Abstract: The emergence of LLMs has ignited a fresh surge of breakthroughs in NLP applications, particularly in domains such as question-answering systems and text generation. As the need for longer context grows, a significant bottleneck in model deployment emerges due to the linear expansion of the Key-Value (KV) cache with the context length. Existing methods primarily rely on various hypotheses, such as sorting the KV cache based on attention scores for replacement or eviction, to compress …

abstract applications arxiv cache context cs.cl deployment domains emergence expansion key linear llm llms model deployment nlp quality quantization question systems text text generation the key type value