Analyzing Semantic Change through Lexical Replacements

arXiv:2404.18570v1 Announce Type: new
Abstract: Modern language models are capable of contextualizing words based on their surrounding context. However, this capability is often compromised due to semantic change that leads to words being used in new, unexpected contexts not encountered during pre-training. In this paper, we model \textit{semantic change} by studying the effect of unexpected contexts introduced by \textit{lexical replacements}. We propose a \textit{replacement schema} where a target word is substituted with lexical replacements of varying relatedness, thus simulating different …

abstract arxiv capability change context cs.cl however language language models leads modern paper pre-training semantic studying through training type words