Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding

arXiv:2402.12374v1 Announce Type: new
Abstract: As the usage of large language models (LLMs) grows, performing efficient inference with these models becomes increasingly important. While speculative decoding has recently emerged as a promising direction for speeding up inference, existing methods are limited in their ability to scale to larger speculation budgets, and adapt to different hyperparameters and hardware. This paper introduces Sequoia, a scalable, robust, and hardware-aware algorithm for speculative decoding. To attain better scalability, Sequoia introduces a dynamic programming algorithm …

abstract adapt arxiv cs.cl decoding hardware inference language language models large language large language models llms robust scalable scale sequoia speculation type usage