[R] General-purpose, long-context autoregressive modeling with Perceiver AR - Deepmind 2022

Paper: [https://arxiv.org/abs/2202.07765](https://arxiv.org/abs/2202.07765)

Deepmind: [https://www.deepmind.com/publications/perceiver-ar-general-purpose-long-context-autoregressive-generation](https://www.deepmind.com/publications/perceiver-ar-general-purpose-long-context-autoregressive-generation)

Abstract:

>Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking. **Perceiver …

ar context deepmind general machinelearning modeling perceiver