Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

arXiv:2401.07851v2 Announce Type: replace
Abstract: To mitigate the high inference latency stemming from autoregressive decoding in Large Language Models (LLMs), Speculative Decoding has emerged as a novel decoding paradigm for LLM inference. In each decoding step, this method first drafts several future tokens efficiently and then verifies them in parallel. Unlike autoregressive decoding, Speculative Decoding facilitates the simultaneous decoding of multiple tokens per step, thereby accelerating inference. This paper presents a comprehensive overview and analysis of this promising decoding paradigm. …

abstract arxiv cs.cl decoding efficiency future inference inference latency language language model language models large language large language model large language models latency llm llms novel paradigm stemming survey tokens type