Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

arXiv:2403.02310v1 Announce Type: new
Abstract: Each LLM serving request goes through two phases. The first is prefill which processes the entire input prompt to produce one output token and the second is decode which generates the rest of output tokens, one-at-a-time. Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt. In contrast, decode iterations have low latency but also low compute utilization because a decode iteration processes only a single token per …

abstract arxiv compute cs.dc cs.lg decode gpu inference latency llm processes prompt request rest serve through token tokens type