Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs

arXiv:2403.00858v1 Announce Type: new
Abstract: Text generation with Large Language Models (LLMs) is known to be memory bound due to the combination of their auto-regressive nature, huge parameter counts, and limited memory bandwidths, often resulting in low token rates. Speculative decoding has been proposed as a solution for LLM inference acceleration. However, since draft models are often unavailable in the modern open-source LLM families, e.g., for Llama 2 7B, training a high-quality draft model is required to enable inference acceleration …

abstract alignment arxiv auto chat combination cs.ai cs.cl cs.lg decoding draft language language models large language large language models llm llms low memory nature solution text text generation token type