Is It Worth the (Environmental) Cost? Limited Evidence for the Benefits of Diachronic Continuous Training. (arXiv:2210.07365v1 [cs.CL])

Language is constantly changing and evolving, leaving language models to
quickly become outdated, both factually and linguistically. Recent research
proposes we continuously update our models using new data. Continuous training
allows us to teach language models about new events and facts and changing
norms. However, continuous training also means continuous costs. We show there
is currently limited evidence for the benefits of continuous training, be it
for the actual downstream performance or the environmental cost. Our results
show continuous training …

arxiv benefits continuous cost environmental evidence training