Exploring Design Choices for Building Language-Specific LLMs

arXiv:2406.14670v1 Announce Type: new
Abstract: Despite rapid progress in large language models (LLMs), their performance on a vast majority of languages remain unsatisfactory. In this paper, we study building language-specific LLMs by adapting monolingual and multilingual LLMs. We conduct systematic experiments on how design choices (base model selection, vocabulary extension, and continued fine-tuning) impact the adapted LLM, both in terms of efficiency (how many tokens are needed to encode the same amount of information) and end task performance. We find …

abstract arxiv building cs.ai cs.cl cs.lg design extension fine-tuning language language models languages large language large language models llms model selection multilingual paper performance progress study tuning type vast