P$^3$LM: Probabilistically Permuted Prophet Language Modeling for Generative Pre-Training. (arXiv:2210.12339v1 [cs.CL])

Conventional autoregressive left-to-right (L2R) sequence generation faces two
issues during decoding: limited to unidirectional target sequence modeling, and
constrained on strong local dependencies. To address the aforementioned
problem, we propose P$^3$LM, a probabilistically permuted prophet language
model, which strengthens the modeling of bidirectional information and long
token dependencies for sequence generation. Specifically, P$^3$LM learns to
generate tokens in permuted order upon an order-aware transformer decoder, as
well as to generate the corresponding future $N$ tokens with a multi-stream
attention mechanism. …

arxiv language modeling pre-training prophet training