Microsoft’s LongRoPE Breaks the Limit of Context Window of LLMs, Extents it to 2 Million Tokens

In a new paper LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens, a Microsoft research team introduces LongRoPE, a pioneering method that extends the context window of pre-trained LLMs to an impressive 2048k tokens while preserving performance at the original short context window.

ai artificial intelligence beyond context context window deep-neural-networks large language model llm llms machine learning machine learning & data science microsoft microsoft research ml paper performance research research team team technology tokens word embedding