Speculative Streaming: Fast LLM Inference without Auxiliary Models

arXiv:2402.11131v1 Announce Type: cross
Abstract: Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target …

abstract application arxiv cs.ai cs.cl cs.lg decoding draft fine-tuning inference language language model llm predictions speed streaming type