On the Robustness of Reading Comprehension Models to Entity Renaming. (arXiv:2110.08555v2 [cs.CL] UPDATED)

Web: http://arxiv.org/abs/2110.08555

We study the robustness of machine reading comprehension (MRC) models to
entity renaming -- do models make more wrong predictions when the same
questions are asked about an entity whose name has been changed? Such failures
imply that models overly rely on entity information to answer questions, and
thus may generalize poorly when facts about the world change or questions are
asked about novel entities. To systematically audit this issue, we present a
pipeline to automatically generate test examples at …

arxiv models on reading robustness