Distillation for Multilingual Information Retrieval

arXiv:2405.00977v1 Announce Type: cross
Abstract: Recent work in cross-language information retrieval (CLIR), where queries and documents are in different languages, has shown the benefit of the Translate-Distill framework that trains a cross-language neural dual-encoder model using translation and distillation. However, Translate-Distill only supports a single document language. Multilingual information retrieval (MLIR), which ranks a multilingual document collection, is harder to train than CLIR because the model must assign comparable relevance scores to documents in different languages. This work extends Translate-Distill …

abstract arxiv benefit cs.cl cs.ir distillation document documents encoder framework however information language languages mlir multilingual queries retrieval trains translate translation type work