General-purpose, long-context autoregressive modeling with Perceiver AR. (arXiv:2202.07765v2 [cs.LG] UPDATED)

Real-world data is high-dimensional: a book, image, or musical performance
can easily contain hundreds of thousands of elements even after compression.
However, the most commonly used autoregressive models, Transformers, are
prohibitively expensive to scale to the number of inputs and layers needed to
capture this long-range structure. We develop Perceiver AR, an autoregressive,
modality-agnostic architecture which uses cross-attention to map long-range
inputs to a small number of latents while also maintaining end-to-end causal
masking. Perceiver AR can directly attend to …

ar arxiv context general lg modeling perceiver