Shallow Cross-Encoders for Low-Latency Retrieval

arXiv:2403.20222v1 Announce Type: cross
Abstract: Transformer-based Cross-Encoders achieve state-of-the-art effectiveness in text retrieval. However, Cross-Encoders based on large transformer models (such as BERT or T5) are computationally expensive and allow for scoring only a small number of documents within a reasonably small latency window. However, keeping search latencies low is important for user satisfaction and energy usage. In this paper, we show that weaker shallow transformer models (i.e., transformers with a limited number of layers) actually perform better than full-scale …

abstract art arxiv bert cs.cl cs.ir documents however latency low retrieval scoring search small state text transformer transformer models type