What's In My Big Data?

arXiv:2310.20707v2 Announce Type: replace-cross
Abstract: Large text corpora are the backbone of language models. However, we have a limited understanding of the content of these corpora, including general statistics, quality, social factors, and inclusion of evaluation data (contamination). In this work, we propose What's In My Big Data? (WIMBD), a platform and a set of sixteen analyses that allow us to reveal and compare the contents of large text corpora. WIMBD builds on two basic capabilities -- count and search …

abstract arxiv big big data cs.cl cs.lg data evaluation general however inclusion language language models platform quality set social statistics text type understanding work