Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora. (arXiv:2110.08534v2 [cs.CL] UPDATED)

Pretrained language models (PTLMs) are typically learned over a large, static
corpus and further fine-tuned for various downstream tasks. However, when
deployed in the real world, a PTLM-based model must deal with data
distributions that deviate from what the PTLM was initially trained on. In this
paper, we study a lifelong language model pretraining challenge where a PTLM is
continually updated so as to adapt to emerging data. Over a domain-incremental
research paper stream and a chronologically-ordered tweet stream, we …

arxiv language language models