DistillSpec: Improving Speculative Decoding via Knowledge Distillation

arXiv:2310.08461v2 Announce Type: replace-cross
Abstract: Speculative decoding (SD) accelerates large language model inference by employing a faster draft model for generating multiple tokens, which are then verified in parallel by the larger target model, resulting in the text generated according to the target model distribution. However, identifying a compact draft model that is well-aligned with the target model is challenging. To tackle this issue, we propose DistillSpec that uses knowledge distillation to better align the draft model with the target …

abstract arxiv compact cs.ai cs.cl cs.lg decoding distillation distribution draft faster generated however improving inference knowledge language language model large language large language model multiple text tokens type via