MuLan: A Study of Fact Mutability in Language Models

arXiv:2404.03036v1 Announce Type: new
Abstract: Facts are subject to contingencies and can be true or false in different circumstances. One such contingency is time, wherein some facts mutate over a given period, e.g., the president of a country or the winner of a championship. Trustworthy language models ideally identify mutable facts as such and process them accordingly. We create MuLan, a benchmark for evaluating the ability of English language models to anticipate time-contingency, covering both 1:1 and 1:N relations. We …

abstract arxiv country cs.cl facts false identify language language models mutability president study true trustworthy type