Inconsistencies in Masked Language Models

arXiv:2301.00068v3 Announce Type: replace
Abstract: Learning to predict masked tokens in a sequence has been shown to be a helpful pretraining objective for powerful language models such as PaLM2. After training, such masked language models (MLMs) can provide distributions of tokens in the masked positions in a sequence. However, this paper shows that distributions corresponding to different masking patterns can demonstrate considerable inconsistencies, i.e., they cannot be derived from a coherent joint distribution when considered together.
This fundamental flaw in …

abstract arxiv cs.ai cs.cl language language models palm2 paper pretraining shows tokens training type