When does word order matter and when doesn't it?

arXiv:2402.18838v1 Announce Type: new
Abstract: Language models (LMs) may appear insensitive to word order changes in natural language understanding (NLU) tasks. In this paper, we propose that linguistic redundancy can explain this phenomenon, whereby word order and other linguistic cues such as case markers provide overlapping and thus redundant information. Our hypothesis is that models exhibit insensitivity to word order when the order provides redundant information, and the degree of insensitivity varies across tasks. We quantify how informative word order …

abstract arxiv case cs.cl information language language models language understanding lms matter natural natural language nlu paper redundancy tasks type understanding word