NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms

arXiv:2402.12261v1 Announce Type: new
Abstract: The performance of Large Language Models (LLMs) degrades from the temporal drift between data used for model training and newer text seen during inference. One understudied avenue of language change causing data drift is the emergence of neologisms -- new word forms -- over time. We create a diverse resource of recent English neologisms by using several popular collection methods. We analyze temporal drift using neologisms by comparing sentences containing new words with near-identical sentences …

abstract arxiv change cs.cl data drift emergence forms inference language language models large language large language models llms neo performance robustness temporal text training type word