[R] LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens - Microsoft 2024

Paper: [https://arxiv.org/abs/2402.13753](https://arxiv.org/abs/2402.13753)

Abstract:

>Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by new token positions, current extended context windows are limited to around 128k tokens. This paper introduces LongRoPE that, for the first time, extends the context window of **pre-trained LLMs to an impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k training lengths, while maintaining performance at …

abstract context context window context windows costs current feature fine-tuning language language models large language large language models llms machinelearning paper token tokens values windows