CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

arXiv:2401.11944v2 Announce Type: replace-cross
Abstract: As the capabilities of large multimodal models (LMMs) continue to advance, evaluating the performance of LMMs emerges as an increasing need. Additionally, there is an even larger gap in evaluating the advanced knowledge and reasoning abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to evaluate LMMs on tasks demanding college-level subject knowledge and deliberate reasoning in a Chinese context. CMMMU is inspired …

abstract advance advanced arxiv benchmark capabilities chinese cs.ai cs.cl cs.cv english gap knowledge large multimodal models lmms massive multimodal multimodal models performance reasoning type understanding