FastSpell: the LangId Magic Spell

arXiv:2404.08345v1 Announce Type: new
Abstract: Language identification is a crucial component in the automated production of language resources, particularly in multilingual and big data contexts. However, commonly used language identifiers struggle to differentiate between similar or closely-related languages. This paper introduces FastSpell, a language identifier that combines fastText (a pre-trained language identifier tool) and Hunspell (a spell checker) with the aim of having a refined second-opinion before deciding which language should be assigned to a text. We provide a description …

abstract arxiv automated big big data cs.cl data however identification language language resources languages magic multilingual paper production resources spell struggle tool type