Representation Deficiency in Masked Language Modeling

arXiv:2302.02060v2 Announce Type: replace-cross
Abstract: Masked Language Modeling (MLM) has been one of the most prominent approaches for pretraining bidirectional text encoders due to its simplicity and effectiveness. One notable concern about MLM is that the special $\texttt{[MASK]}$ symbol causes a discrepancy between pretraining data and downstream data as it is present only in pretraining but not in fine-tuning. In this work, we offer a new perspective on the consequence of such a discrepancy: We demonstrate empirically and theoretically that …

abstract arxiv cs.cl cs.lg data language modeling pretraining representation simplicity text type