MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria

arXiv:2311.13951v2 Announce Type: replace
Abstract: Multimodal large language models (MLLMs) (e.g., GPT-4V, LLaVA, and Claude-3) have broadened the scope of AI applications. Yet, evaluating their performance presents a significant challenge owing to the inherently subjective nature of tasks that do not yield clear-cut solutions especially for those open-ended queries. Existing automatic evaluation methodologies are mainly limited in evaluating objective queries without considering real-world user experiences, inadequately addressing the nuances of creative and associative multimodal tasks. In our paper, we propose …

arxiv cs.cl llms mllm multimodal multimodal llms per sample type