MaLA-500: Massive Language Adaptation of Large Language Models

arXiv:2401.13303v2 Announce Type: replace
Abstract: Large language models (LLMs) have advanced the state of the art in natural language processing. However, their predominant design for English or a limited set of languages creates a substantial gap in their effectiveness for low-resource languages. To bridge this gap, we introduce MaLA-500, a novel large language model designed to cover an extensive range of 534 languages. To train MaLA-500, we employ vocabulary extension and continued pretraining on LLaMA 2 with Glot500-c. Our intrinsic …

arxiv cs.cl language language models large language large language models massive type