E^2-LLM: Efficient and Extreme Length Extension of Large Language Models

arXiv:2401.06951v3 Announce Type: replace
Abstract: Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources. Existing long-context extension methods usually need additional training procedures to support corresponding long-context windows, where the long-context training data (e.g., 32k) is needed, and high GPU training costs are assumed. To address the aforementioned issues, we propose an Efficient and Extreme length extension method for Large Language Models, called E 2 -LLM, with only one training procedure and …

abstract arxiv context context windows costs cs.ai cs.cl data extension gpu gpu resources language language models large language large language models llm llms resources support training training costs training data training llms type windows