Meta-Task Prompting Elicits Embedding from Large Language Models

arXiv:2402.18458v1 Announce Type: new
Abstract: In this work, we introduce a new unsupervised embedding method, Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL), for generating high-quality sentence embeddings from Large Language Models (LLMs) without the need for model fine-tuning or task-specific engineering. Leveraging meta-task prompting, MetaEOL guides LLMs to produce embeddings through a series of carefully designed prompts that address multiple representational aspects. Our comprehensive experiments demonstrate that embeddings averaged from various meta-tasks yield competitive performance on Semantic Textual Similarity (STS) …

abstract arxiv cs.cl embedding embeddings engineering fine-tuning guides language language models large language large language models llms meta model fine-tuning prompting quality through type unsupervised word work