Optimal Block-Level Draft Verification for Accelerating Speculative Decoding

arXiv:2403.10444v1 Announce Type: new
Abstract: Speculative decoding has shown to be an effective method for lossless acceleration of large language models (LLMs) during inference. In each iteration, the algorithm first uses a smaller model to draft a block of tokens. The tokens are then verified by the large model in parallel and only a subset of tokens will be kept to guarantee that the final output follows the distribution of the large model. In all of the prior speculative decoding …

abstract algorithm arxiv block cs.cl cs.ds cs.it cs.lg decoding draft inference iteration language language models large language large language models llms math.it the algorithm tokens type verification