Training and Inference on Any-Order Autoregressive Models the Right Way. (arXiv:2205.13554v2 [cs.LG] UPDATED)

Conditional inference on arbitrary subsets of variables is a core problem in
probabilistic inference with important applications such as masked language
modeling and image inpainting. In recent years, the family of Any-Order
Autoregressive Models (AO-ARMs) -- closely related to popular models such as
BERT and XLNet -- has shown breakthrough performance in arbitrary conditional
tasks across a sweeping range of domains. But, in spite of their success, in
this paper we identify significant improvements to be made to previous
formulations …

arxiv autoregressive models inference training