Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference

arXiv:2403.09636v1 Announce Type: new
Abstract: Transformers have emerged as the backbone of large language models (LLMs). However, generation remains inefficient due to the need to store in memory a cache of key-value representations for past tokens, whose size scales linearly with the input sequence length and batch size. As a solution, we propose Dynamic Memory Compression (DMC), a method for on-line key-value cache compression at inference time. Most importantly, the model learns to apply different compression rates in different heads …

abstract arxiv cache compression cs.cl dynamic however inference key language language models large language large language models llms memory solution store tokens transformers type value