Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval

arXiv:2401.11248v2 Announce Type: replace-cross
Abstract: Masked auto-encoder pre-training has emerged as a prevalent technique for initializing and enhancing dense retrieval systems. It generally utilizes additional Transformer decoder blocks to provide sustainable supervision signals and compress contextual information into dense representations. However, the underlying reasons for the effectiveness of such a pre-training technique remain unclear. The usage of additional Transformer-based decoders also incurs significant computational costs. In this study, we aim to shed light on this issue by revealing that masked …

abstract arxiv auto bag cs.cl cs.ir decoder encoder however information prediction pre-training retrieval supervision sustainable systems training transformer transformer decoder type word