MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification

arXiv:2311.09761v2 Announce Type: replace-cross
Abstract: We introduce MAFALDA, a benchmark for fallacy classification that merges and unites previous fallacy datasets. It comes with a taxonomy that aligns, refines, and unifies existing classifications of fallacies. We further provide a manual annotation of a part of the dataset together with manual explanations for each annotation. We propose a new annotation scheme tailored for subjective NLP tasks, and a new evaluation method designed to handle subjectivity. We then evaluate several language models under …

abstract annotation arxiv benchmark classification cs.ai cs.cl cs.lg dataset datasets detection part study taxonomy together type