No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrieval. (arXiv:2206.02873v3 [cs.IR] UPDATED)

Recent work has shown that small distilled language models are strong
competitors to models that are orders of magnitude larger and slower in a wide
range of information retrieval tasks. This has made distilled and dense models,
due to latency constraints, the go-to choice for deployment in real-world
retrieval applications. In this work, we question this practice by showing that
the number of parameters and early query-document interaction play a
significant role in the generalization ability of retrieval models. Our …

arxiv distillation retrieval