Accelerating Speculative Decoding using Dynamic Speculation Length

arXiv:2405.04304v1 Announce Type: new
Abstract: Speculative decoding is a promising method for reducing the inference latency of large language models. The effectiveness of the method depends on the speculation length (SL) - the number of tokens generated by the draft model at each iteration. The vast majority of speculative decoding approaches use the same SL for all iterations. In this work, we show that this practice is suboptimal. We introduce DISCO, a DynamIc SpeCulation length Optimization method that uses a …

abstract arxiv cs.cl decoding draft dynamic generated inference inference latency iteration language language models large language large language models latency speculation tokens type vast