Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting

arXiv:2404.18911v1 Announce Type: cross
Abstract: Speculative decoding has demonstrated its effectiveness in accelerating the inference of large language models while maintaining a consistent sampling distribution. However, the conventional approach of training a separate draft model to achieve a satisfactory token acceptance rate can be costly. Drawing inspiration from early exiting, we propose a novel self-speculative decoding framework \emph{Kangaroo}, which uses a fixed shallow sub-network as a self-draft model, with the remaining layers serving as the larger target model. We train …

arxiv cs.cl cs.lg decoding type via