DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token Sampling

arXiv:2405.00888v1 Announce Type: new
Abstract: Traditional language models operate autoregressively, i.e., they predict one token at a time. Rapid explosion in model sizes has resulted in high inference times. In this work, we propose DynaMo, a suite of multi-token prediction language models that reduce net inference times. Our models $\textit{dynamically}$ predict multiple tokens based on their confidence in the predicted joint probability distribution. We propose a lightweight technique to train these models, leveraging the weights of traditional autoregressive counterparts. Moreover, …

abstract arxiv cs.cl dynamic inference language language model language models prediction reduce sampling token type work