CacheGen: Fast Context Loading for Language Model Applications via KV Cache Streaming

arXiv:2310.07240v2 Announce Type: replace-cross
Abstract: As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge or user-specific information. Yet using long contexts poses a challenge for responsive LLM systems, as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over …

abstract applications arxiv cache challenge context cs.lg cs.ni domain domain knowledge generated information inputs knowledge language language model language models large language large language models llm llms loading nothing responsive streaming systems tasks type via