[R] Better & Faster Large Language Models via Multi-token Prediction

**Paper**: [https://arxiv.org/abs/2404.19737](https://arxiv.org/abs/2404.19737)

**Abstract**:

>Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict *multiple* future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. Considering multi-token prediction as an auxiliary training task, we …

abstract efficiency future gpt independent language language models large language large language models llama loss machinelearning multiple next prediction results sample token tokens training work