Examining Single Sentence Label Leakage in Natural Language Inference Datasets. (arXiv:2112.09237v3 [cs.CL] UPDATED)

Many believe human-level natural language inference (NLI) has already been
achieved. In reality, modern NLI benchmarks have serious flaws, rendering
progress questionable. Chief among them is the problem of single sentence label
leakage, where spurious correlations and biases in datasets enable the accurate
prediction of a sentence pair relation from only a single sentence, something
that should in principle be impossible. This leakage enables models to cheat
rather than learn the desired reasoning capabilities, and hasn't gone away
since its …

arxiv datasets inference language natural natural language