Simple and Scalable Strategies to Continually Pre-train Large Language Models

arXiv:2403.08763v1 Announce Type: new
Abstract: Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of …

abstract arxiv compute cs.ai cs.cl cs.lg data distribution however language language models large language large language models llms process saving scalable shift simple solution strategies tokens train training type