On the Role of Bidirectionality in Language Model Pre-Training. (arXiv:2205.11726v1 [cs.CL])

Prior work on language model pre-training has explored different
architectures and learning objectives, but differences in data, hyperparameters
and evaluation make a principled comparison difficult. In this work, we focus
on bidirectionality as a key factor that differentiates existing approaches,
and present a comprehensive study of its role in next token prediction, text
infilling, zero-shot priming and fine-tuning. We propose a new framework that
generalizes prior approaches, including fully unidirectional models like GPT,
fully bidirectional models like BERT, and hybrid …

arxiv language language model pre-training role training