LongEmbed: Extending Embedding Models for Long Context Retrieval

arXiv:2404.12096v1 Announce Type: cross
Abstract: Embedding models play a pivot role in modern NLP applications such as IR and RAG. While the context limit of LLMs has been pushed beyond 1 million tokens, embedding models are still confined to a narrow context window not exceeding 8k tokens, refrained from application scenarios requiring long inputs such as legal contracts. This paper explores context window extension of existing embedding models, pushing the limit to 32k without requiring additional training. First, we examine …

abstract application applications arxiv beyond context context window cs.cl cs.lg embedding embedding models llms modern narrow nlp pivot rag retrieval role tokens type