Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference

arXiv:2403.09054v1 Announce Type: new
Abstract: Transformers have emerged as the underpinning architecture for Large Language Models (LLMs). In generative language models, the inference process involves two primary phases: prompt processing and token generation. Token generation, which constitutes the majority of the computational workload, primarily entails vector-matrix multiplications and interactions with the Key-Value (KV) Cache. This phase is constrained by memory bandwidth due to the overhead of transferring weights and KV cache values from the memory system to the computing units. …

abstract architecture arxiv cache computational cs.ai cs.ar cs.lg generative inference key language language models large language large language models llms matrix process processing prompt through token tokens transformers type vector