Uncovering More Shallow Heuristics: Probing the Natural Language Inference Capacities of Transformer-Based Pre-Trained Language Models Using Syllogistic Patterns. (arXiv:2201.07614v1 [cs.CL])

In this article, we explore the shallow heuristics used by transformer-based
pre-trained language models (PLMs) that are fine-tuned for natural language
inference (NLI). To do so, we construct or own dataset based on syllogistic,
and we evaluate a number of models' performance on our dataset. We find
evidence that the models rely heavily on certain shallow heuristics, picking up
on symmetries and asymmetries between premise and hypothesis. We suggest that
the lack of generalization observable in our study, which is …

arxiv language language models natural natural language transformer