SubGen: Token Generation in Sublinear Time and Memory

Despite the significant success of large language models (LLMs), their extensive memory requirements pose challenges for deploying them in long-context token generation. The substantial memory footprint of LLM decoders arises from the necessity to store all previous tokens in the attention module, a requirement imposed by key-value (KV) caching. In this work, our focus is on developing an efficient compression technique for the KV cache. Empirical evidence indicates a significant clustering tendency within key embeddings in the attention module. Building …

attention caching challenges context cs.ai cs.ds cs.lg key language language models large language large language models llm llms memory requirements store success them token tokens value work