LLMs for Generating and Evaluating Counterfactuals: A Comprehensive Study

arXiv:2405.00722v1 Announce Type: new
Abstract: As NLP models become more complex, understanding their decisions becomes more crucial. Counterfactuals (CFs), where minimal changes to inputs flip a model's prediction, offer a way to explain these models. While Large Language Models (LLMs) have shown remarkable performance in NLP tasks, their efficacy in generating high-quality CFs remains uncertain. This work fills this gap by investigating how well LLMs generate CFs for two NLU tasks. We conduct a comprehensive comparison of several common LLMs, …

abstract arxiv become cs.ai cs.cl decisions inputs language language models large language large language models llms nlp nlp models performance prediction study tasks type understanding while