REST: Retrieval-Based Speculative Decoding

arXiv:2311.08252v2 Announce Type: replace-cross
Abstract: We introduce Retrieval-Based Speculative Decoding (REST), a novel algorithm designed to speed up language model generation. The key insight driving the development of REST is the observation that the process of text generation often includes certain common phases and patterns. Unlike previous methods that rely on a draft language model for speculative decoding, REST harnesses the power of retrieval to generate draft tokens. This method draws from the reservoir of existing knowledge, retrieving and employing …

arxiv cs.ai cs.cl cs.ir cs.lg decoding rest retrieval type