Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models

arXiv:2405.02287v1 Announce Type: cross
Abstract: We introduce Vibe-Eval: a new open benchmark and framework for evaluating multimodal chat models. Vibe-Eval consists of 269 visual understanding prompts, including 100 of hard difficulty, complete with gold-standard responses authored by experts. Vibe-Eval is open-ended and challenging with dual objectives: (i) vibe checking multimodal chat models for day-to-day tasks and (ii) rigorously testing and probing the capabilities of present frontier models. Notably, our hard set contains >50% questions that all frontier models answer incorrectly. …

arxiv cs.ai cs.cl cs.cv evaluation language language models measuring multimodal multimodal language models progress type