Benchmark Transparency: Measuring the Impact of Data on Evaluation

arXiv:2404.00748v1 Announce Type: new
Abstract: In this paper we present an exploratory research on quantifying the impact that data distribution has on the performance and evaluation of NLP models. We propose an automated framework that measures the data point distribution across 6 different dimensions: ambiguity, difficulty, discriminability, length, noise, and perplexity.
We use disproportional stratified sampling to measure how much the data distribution affects absolute (Acc/F1) and relative (Rank) model performance. We experiment on 2 different datasets (SQUAD and MNLI) …

abstract arxiv automated benchmark cs.ai cs.cl data dimensions distribution evaluation exploratory framework impact measuring nlp nlp models noise paper performance perplexity research transparency type