Training and Inference on Any-Order Autoregressive Models the Right Way. (arXiv:2205.13554v1 [cs.LG])

Conditional inference on arbitrary subsets of variables is a core problem in
probabilistic inference with important applications such as masked language
modeling and image inpainting. In recent years, the family of Any-Order
Autoregressive Models (AO-ARMs) -- which includes popular models such as XLNet
-- has shown breakthrough performance in arbitrary conditional tasks across a
sweeping range of domains. But, in spite of their success, in this paper we
identify significant improvements to be made to previous formulations of
AO-ARMs. First, …

arxiv autoregressive models inference training