Data Engineering for Scaling Language Models to 128K Context

arXiv:2402.10171v1 Announce Type: new
Abstract: We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular \textit{the ability to utilize information at arbitrary input locations}, is a capability that is mostly already acquired through large-scale pretraining, and that this capability can be readily extended to contexts substantially longer than seen during training~(e.g., 4K to 128K) through lightweight continual pretraining on appropriate data mixture. …

128k context abstract acquired arxiv capability context continual cs.ai cs.cl data data engineering engineering focus information language language models locations modeling pretraining recipe scaling study through type