Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval

arXiv:2311.05800v2 Announce Type: replace-cross
Abstract: There has been limited success for dense retrieval models in multilingual retrieval, due to uneven and scarce training data available across multiple languages. Synthetic training data generation is promising (e.g., InPars or Promptagator), but has been investigated only for English. Therefore, to study model capabilities across both cross-lingual and monolingual retrieval tasks, we develop SWIM-IR, a synthetic retrieval training dataset containing 33 (high to very-low resource) languages for fine-tuning multilingual dense retrievers without requiring any …

arxiv cs.ai cs.cl cs.ir data languages llms multilingual retrieval training training data type