Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval

arXiv:2404.04163v1 Announce Type: cross
Abstract: This study investigates the existence of positional biases in Transformer-based models for text representation learning, particularly in the context of web document retrieval. We build on previous research that demonstrated loss of information in the middle of input sequences for causal language models, extending it to the domain of representation learning. We examine positional biases at various stages of training for an encoder-decoder model, including language model pre-training, contrastive pre-training, and contrastive fine-tuning. Experiments with …

abstract arxiv biases build causal context cs.cl cs.ir document documents embed information language language models loss representation representation learning research retrieval study text transformer transformer-based models type web