Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models

arXiv:2311.08106v2 Announce Type: replace
Abstract: The dynamic nature of knowledge in an ever-changing world presents challenges for language models trained on static data; the model in the real world often requires not only acquiring new knowledge but also overwriting outdated information into updated ones. To study the ability of language models for these time-dependent dynamics in human language, we introduce a novel task, EvolvingQA, a temporally evolving question-answering benchmark designed for training and evaluating LMs on an evolving Wikipedia database. …

abstract arxiv challenges cs.cl data dynamic evaluation ever information knowledge language language models nature ones study type world