A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets

arXiv:2403.03909v1 Announce Type: new
Abstract: Typologically diverse benchmarks are increasingly created to track the progress achieved in multilingual NLP. Linguistic diversity of these data sets is typically measured as the number of languages or language families included in the sample, but such measures do not consider structural properties of the included languages. In this paper, we propose assessing linguistic diversity of a data set against a reference language sample as a means of maximising linguistic diversity in the long run. …

abstract arxiv benchmarks comparison cs.cl data data sets diverse diversity families language languages multilingual nlp progress sample type