MultiContrievers: Analysis of Dense Retrieval Representations

arXiv:2402.15925v1 Announce Type: new
Abstract: Dense retrievers compress source documents into (possibly lossy) vector representations, yet there is little analysis of what information is lost versus preserved, and how it affects downstream tasks. We conduct the first analysis of the information captured by dense retrievers compared to the language models they are based on (e.g., BERT versus Contriever). We use 25 MultiBert checkpoints as randomized initialisations to train MultiContrievers, a set of 25 contriever models. We test whether specific pieces …

abstract analysis arxiv cs.ai cs.cl cs.ir documents information language language models lost retrieval tasks the information type vector