Natural Language Processing for Dialects of a Language: A Survey

arXiv:2401.05632v2 Announce Type: replace
Abstract: State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language. Motivated by the performance degradation of NLP models for dialectic datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches. We describe a wide range of …

abstract art arxiv cs.cl datasets evaluation language language processing massive natural natural language natural language processing nlp nlp models performance processing report state survey training type