RelayAttention for Efficient Large Language Model Serving with Long System Prompts

arXiv:2402.14808v1 Announce Type: new
Abstract: Practical large language model (LLM) services may involve a long system prompt, which specifies the instructions, examples, and knowledge documents of the task and is reused across numerous requests. However, the long system prompt causes throughput/latency bottlenecks as the cost of generating the next token grows w.r.t. the sequence length. This paper aims to improve the efficiency of LLM services that involve long system prompts. Our key observation is that handling these system prompts requires …

abstract arxiv bottlenecks cost cs.cl documents examples knowledge language language model large language large language model latency llm next practical prompt prompts services type