CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

arXiv:2401.14011v3 Announce Type: replace
Abstract: Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question …

arxiv benchmark chinese cs.ai cs.cl cs.mm modal multi-modal question reasoning type understanding