Repetition Improves Language Model Embeddings

arXiv:2402.15449v1 Announce Type: cross
Abstract: Recent approaches to improving the extraction of text embeddings from autoregressive large language models (LLMs) have largely focused on improvements to data, backbone pretrained language models, or improving task-differentiation via instructions. In this work, we address an architectural limitation of autoregressive models: token embeddings cannot contain information from tokens that appear later in the input. To address this limitation, we propose a simple approach, "echo embeddings," in which we repeat the input twice in context …

abstract arxiv autoregressive models cs.cl cs.lg data differentiation embeddings extraction improvements information language language model language models large language large language models llms text token tokens type via work