{\sigma}-GPTs: A New Approach to Autoregressive Models

arXiv:2404.09562v1 Announce Type: new
Abstract: Autoregressive models, such as the GPT family, use a fixed order, usually left-to-right, to generate sequences. However, this is not a necessity. In this paper, we challenge this assumption and show that by simply adding a positional encoding for the output, this order can be modulated on-the-fly per-sample which offers key advantageous properties. It allows for the sampling of and conditioning on arbitrary subsets of tokens, and it also allows sampling in one shot multiple …

abstract arxiv autoregressive autoregressive models challenge cs.ai cs.lg encoding family fly generate gpt gpts however paper per positional encoding sample show type