An Improved Traditional Chinese Evaluation Suite for Foundation Model

arXiv:2403.01858v1 Announce Type: new
Abstract: We present TMMLU+, a comprehensive dataset designed for the Traditional Chinese massive multitask language understanding dataset. TMMLU+ is a multiple-choice question-answering dataset with 66 subjects from elementary to professional level. Compared to its predecessor, TMMLU, TMMLU+ is six times larger and boasts a more balanced subject distribution. We included benchmark results in TMMLU+ from closed-source models and 24 open-weight Chinese large language models of parameters ranging from 1.8B to 72B. Our findings reveal that Traditional …

abstract arxiv chinese cs.cl dataset elementary evaluation foundation foundation model language language understanding massive multiple professional question six type understanding