NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural

arXiv:2403.01817v1 Announce Type: new
Abstract: Indonesia's linguistic landscape is remarkably diverse, encompassing over 700 languages and dialects, making it one of the world's most linguistically rich nations. This diversity, coupled with the widespread practice of code-switching and the presence of low-resource regional languages, presents unique challenges for modern pre-trained language models. In response to these challenges, we developed NusaBERT, building upon IndoBERT by incorporating vocabulary expansion and leveraging a diverse multilingual corpus that includes regional languages and dialects. Through rigorous …

abstract arxiv challenges code cs.cl diverse diversity indonesia landscape language language models languages low making modern multilingual practice regional teaching type world