RetroMAE: Pre-training Retrieval-oriented Transformers via Masked Auto-Encoder. (arXiv:2205.12035v1 [cs.CL])

Pre-trained models have demonstrated superior power on many important tasks.
However, it is still an open problem of designing effective pre-training
strategies so as to promote the models' usability on dense retrieval. In this
paper, we propose a novel pre-training framework for dense retrieval based on
the Masked Auto-Encoder, known as RetroMAE. Our proposed framework is
highlighted for the following critical designs: 1) a MAE based pre-training
workflow, where the input sentence is polluted on both encoder and decoder side …

arxiv encoder pre-training retrieval training transformers