Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons

arXiv:2403.17760v1 Announce Type: new
Abstract: In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias. We then create further challenging sub-tasks in an effort to explain this failure. From a Computational Linguistics perspective, we identify …

abstract arxiv challenge cs.cl dataset language language models large language large language models nlp paper perspective perspectives possibility small them type