Modeling Orthographic Variation Improves NLP Performance for Nigerian Pidgin

arXiv:2404.18264v1 Announce Type: new
Abstract: Nigerian Pidgin is an English-derived contact language and is traditionally an oral language, spoken by approximately 100 million people. No orthographic standard has yet been adopted, and thus the few available Pidgin datasets that exist are characterised by noise in the form of orthographic variations. This contributes to under-performance of models in critical NLP tasks. The current work is the first to describe various types of orthographic variations commonly found in Nigerian Pidgin texts, and …

abstract arxiv cs.ai cs.cl datasets english form language modeling nlp noise people performance spoken standard type variation