LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

arXiv:2404.05961v1 Announce Type: new
Abstract: Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive …

abstract art arxiv benchmarks community cs.ai cs.cl decoder embedding language language models large language large language models llms nlp simple state state-of-the-art models tasks text text embedding type unsupervised work