[R] MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Paper: [https://arxiv.org/abs/2311.16502](https://arxiv.org/abs/2311.16502)

Blog: [https://mmmu-benchmark.github.io/](https://mmmu-benchmark.github.io/)

Abstract:

>We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly heterogeneous image types, such as charts, diagrams, maps, tables, …

abstract art benchmark business college core design engineering exams health humanities knowledge machinelearning massive medicine multimodal multimodal models questions reasoning science six social social science tasks tech