ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic

arXiv:2402.12840v1 Announce Type: new
Abstract: The focus of language model evaluation has transitioned towards reasoning and knowledge-intensive tasks, driven by advancements in pretraining large models. While state-of-the-art models are partially trained on large Arabic texts, evaluating their performance in Arabic remains challenging due to the limited availability of relevant datasets. To bridge this gap, we present ArabicMMLU, the first multi-task language understanding benchmark for Arabic language, sourced from school exams across diverse educational levels in different countries spanning North Africa, …

abstract arabic art arxiv availability bridge cs.cl datasets evaluation focus knowledge language language model language understanding large models massive performance pretraining reasoning state state-of-the-art models tasks type understanding