VALUE: Understanding Dialect Disparity in NLU. (arXiv:2204.03031v1 [cs.CL])

English Natural Language Understanding (NLU) systems have achieved great
performances and even outperformed humans on benchmarks like GLUE and
SuperGLUE. However, these benchmarks contain only textbook Standard American
English (SAE). Other dialects have been largely overlooked in the NLP
community. This leads to biased and inequitable NLU systems that serve only a
sub-population of speakers. To understand disparities in current models and to
facilitate more dialect-competent NLU systems, we introduce the VernAcular
Language Understanding Evaluation (VALUE) benchmark, a challenging variant …

arxiv nlu value