LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

arXiv:2402.13753v1 Announce Type: new
Abstract: Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by new token positions, current extended context windows are limited to around 128k tokens. This paper introduces LongRoPE that, for the first time, extends the context window of pre-trained LLMs to an impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k training lengths, while maintaining …

abstract arxiv beyond context context window context windows costs cs.cl current feature fine-tuning language language models large language large language models llm llms paper token tokens type values windows