EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries

arXiv:2402.13372v1 Announce Type: new
Abstract: While Large Language Models (LLMs) excel at the Winograd Schema Challenge (WSC), a coreference resolution task testing common-sense reasoning through pronoun disambiguation, they struggle with instances that feature minor alterations or rewording. To address this, we introduce EvoGrad, an open-source platform that harnesses a human-in-the-loop approach to create a dynamic dataset tailored to such altered WSC instances. Leveraging ChatGPT's capabilities, we expand our task instances from 182 to 3,691, setting a new benchmark for diverse …

abstract arxiv challenge cs.cl dynamic excel feature human instances language language models large language large language models llms platform reasoning schema sense struggle testing through type