Multi-Token Prediction (forget next token LLM?)

Meta published a new method of multi-token prediction for autoregressive transformer models (LLMs).

Additional heads perform in parallel token predictions. Benchmark data investigated and a special session for my green grasshoppers!

Instead of sequentially predicting the next token based on previously observed tokens, this architecture employs multiple output heads that operate in parallel from a shared trunk—the main body of the model which processes the input and generates a common latent representation. Each output head predicts a different future token …

architecture autoregressive benchmark data green llm llms meta multiple next prediction predictions session token tokens transformer transformer models