CEval: A Benchmark for Evaluating Counterfactual Text Generation

arXiv:2404.17475v1 Announce Type: new
Abstract: Counterfactual text generation aims to minimally change a text, such that it is classified differently. Judging advancements in method development for counterfactual text generation is hindered by a non-uniform usage of data sets and metrics in related work. We propose CEval, a benchmark for comparing counterfactual text generation methods. CEval unifies counterfactual and text quality metrics, includes common counterfactual datasets with human annotations, standard baselines (MICE, GDBA, CREST) and the open-source language model LLAMA-2. Our …

abstract arxiv benchmark change counterfactual cs.ai cs.cl data data sets development metrics text text generation type uniform usage work