KMMLU: Measuring Massive Multitask Language Understanding in Korean

arXiv:2402.11548v1 Announce Type: new
Abstract: We propose KMMLU, a new Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. Unlike previous Korean benchmarks that are translated from existing English benchmarks, KMMLU is collected from original Korean exams, capturing linguistic and cultural aspects of the Korean language. We test 26 publically available and proprietary LLMs, identifying significant room for improvement. The best publicly available model achieves 50.54% on KMMLU, far below the average human performance …

abstract arxiv benchmark benchmarks cs.cl english exams expert humanities language language understanding massive measuring multiple questions stem translated type understanding