Towards Continual Knowledge Learning of Language Models. (arXiv:2110.03215v4 [cs.CL] UPDATED)

Large Language Models (LMs) are known to encode world knowledge in their
parameters as they pretrain on a vast amount of web corpus, which is often
utilized for performing knowledge-dependent downstream tasks such as question
answering, fact-checking, and open dialogue. In real-world scenarios, the world
knowledge stored in the LMs can quickly become outdated as the world changes,
but it is non-trivial to avoid catastrophic forgetting and reliably acquire new
knowledge while preserving invariant knowledge. To push the community towards …

arxiv continual knowledge language language models learning