Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics

Language models (LMs) have been argued to overlap substantially with human beings in grammaticality judgment tasks. But when humans systematically make errors in language processing, should we expect LMs to behave like cognitive models of language and mimic human behavior? We answer this question by investigating LMs' more subtle judgments associated with "language illusions" -- sentences that are vague in meaning, implausible, or ungrammatical but receive unexpectedly high acceptability judgments by humans. We looked at three illusions: the comparative illusion …

behavior beings cognitive cs.cl errors expect human humans judgment language language models language processing lms processing question semantics syntax tasks