Efficient LLM Inference with Kcache

arXiv:2404.18057v1 Announce Type: new
Abstract: Large Language Models(LLMs) have had a profound impact on AI applications, particularly in the domains of long-text comprehension and generation. KV Cache technology is one of the most widely used techniques in the industry. It ensures efficient sequence generation by caching previously computed KV states. However, it also introduces significant memory overhead. We discovered that KV Cache is not necessary and proposed a novel KCache technique to alleviate the memory bottleneck issue during the LLMs …

abstract ai applications applications arxiv cache caching cs.cl domains however impact industry inference language language models large language large language models llm llms technology text type