WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation. (arXiv:2201.05955v4 [cs.CL] UPDATED)

A recurring challenge of crowdsourcing NLP datasets at scale is that human
writers often rely on repetitive patterns when crafting examples, leading to a
lack of linguistic diversity. We introduce a novel approach for dataset
creation based on worker and AI collaboration, which brings together the
generative strength of language models and the evaluative strength of humans.
Starting with an existing dataset, MultiNLI for natural language inference
(NLI), our approach uses dataset cartography to automatically identify examples
that demonstrate challenging …

ai collaboration arxiv collaboration dataset inference language natural natural language