Accelerating Production LLMs with Combined Token/Embedding Speculators

arXiv:2404.19124v1 Announce Type: new
Abstract: This technical report describes the design and training of novel speculative decoding draft models, for accelerating the inference speeds of large language models in a production environment. By conditioning draft predictions on both context vectors and sampled tokens, we can train our speculators to efficiently predict high-quality n-grams, which the base model then accepts or rejects. This allows us to effectively predict multiple tokens per inference forward pass, accelerating wall-clock inference speeds of highly optimized …

abstract arxiv context cs.cl decoding design draft embedding environment inference language language models large language large language models llms novel predictions production quality report technical token tokens train training type vectors