What Drives Performance in Multilingual Language Models?

arXiv:2404.19159v1 Announce Type: new
Abstract: This study investigates the factors influencing the performance of multilingual large language models (MLLMs) across diverse languages. We study 6 MLLMs, including masked language models, autoregressive models, and instruction-tuned LLMs, on the SIB-200 dataset, a topic classification dataset encompassing 204 languages. Our analysis considers three scenarios: ALL languages, SEEN languages (present in the model's pretraining data), and UNSEEN languages (not present or documented in the model's pretraining data in any meaningful way). We examine the …

abstract analysis arxiv autoregressive autoregressive models classification cs.cl dataset diverse instruction-tuned language language models languages large language large language models llms mllms multilingual performance study type