Set the Clock: Temporal Alignment of Pretrained Language Models

arXiv:2402.16797v1 Announce Type: new
Abstract: Language models (LMs) are trained on web text originating from many points in time and, in general, without any explicit temporal grounding. This work investigates the temporal chaos of pretrained LMs and explores various methods to align their internal knowledge to a target time, which we call "temporal alignment." To do this, we first automatically construct a dataset containing 20K time-sensitive questions and their answers for each year from 2000 to 2023. Based on this …

abstract alignment arxiv chaos cs.cl general knowledge language language models lms set temporal text type web work