Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT. (arXiv:2209.10538v1 [cs.CL])

Both humans and neural language models are able to perform subject-verb
number agreement (SVA). In principle, semantics shouldn't interfere with this
task, which only requires syntactic knowledge. In this work we test whether
meaning interferes with this type of agreement in English in syntactic
structures of various complexities. To do so, we generate both semantically
well-formed and nonsensical items. We compare the performance of BERT-base to
that of humans, obtained with a psycholinguistic online crowdsourcing
experiment. We find that BERT …

arxiv bert error humans patterns