Geographically-Informed Language Identification

arXiv:2403.09892v1 Announce Type: new
Abstract: This paper develops an approach to language identification in which the set of languages considered by the model depends on the geographic origin of the text in question. Given that many digital corpora can be geo-referenced at the country level, this paper formulates 16 region-specific models, each of which contains the languages expected to appear in countries within that region. These regional models also each include 31 widely-spoken international languages in order to ensure coverage …

abstract arxiv country cs.cl digital geo identification language languages paper question set text type