Gecko: Versatile Text Embeddings Distilled from Large Language Models

arXiv:2403.20327v1 Announce Type: new
Abstract: We present Gecko, a compact and versatile text embedding model. Gecko achieves strong retrieval performance by leveraging a key idea: distilling knowledge from large language models (LLMs) into a retriever. Our two-step distillation process begins with generating diverse, synthetic paired data using an LLM. Next, we further refine the data quality by retrieving a set of candidate passages for each query, and relabeling the positive and hard negative passages using the same LLM. The effectiveness …

abstract arxiv compact cs.ai cs.cl data distillation diverse embedding embeddings gecko key knowledge language language models large language large language models llm llms next performance process retrieval synthetic text text embedding type