Transforming LLMs into Cross-modal and Cross-lingual RetrievalSystems

arXiv:2404.01616v1 Announce Type: new
Abstract: Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval systems project queries and documents into the same embedding space and have demonstrated their success in retrieval and bi-text mining. To match speech and text in many languages, we propose using LLMs to initialize multi-modal DE retrieval systems. Unlike traditional methods, our system doesn't require …

abstract arxiv beyond cross-lingual cs.cl cs.ir cs.sd data documents eess.as embedding encoder language language models languages large language large language models llms modal project queries retrieval space speech success systems text type