SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

arXiv:2305.09781v4 Announce Type: replace-cross
Abstract: This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind SpecInfer is leveraging small speculative models to predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified against the LLM in parallel using a novel tree-based parallel …

arxiv cs.cl cs.dc cs.lg generative inference language language model large language large language model tree type verification