Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction

arXiv:2404.08509v1 Announce Type: cross
Abstract: Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a …

abstract ai applications applications arxiv autoregressive cs.cl cs.dc cs.lg domains driving exploit generative generative models however inference interactive language language models large language large language models llm llms nature prediction serve systems type