Towards Fast Inference: Exploring and Improving Blockwise Parallel Drafts

arXiv:2404.09221v1 Announce Type: cross
Abstract: Despite the remarkable strides made by autoregressive language models, their potential is often hampered by the slow inference speeds inherent in sequential token generation. Blockwise parallel decoding (BPD) was proposed by Stern et al. (2018) as a way to improve inference speed of language models. In this paper, we make two contributions to understanding and improving BPD drafts. We first offer an analysis of the token distributions produced by the BPD prediction heads. Secondly, we …

abstract arxiv autoregressive cs.ai cs.cl cs.lg decoding improving inference language language models speed token type