Uncertainty-Aware Evaluation for Vision-Language Models

arXiv:2402.14418v1 Announce Type: new
Abstract: Vision-Language Models like GPT-4, LLaVA, and CogVLM have surged in popularity recently due to their impressive performance in several vision-language tasks. Current evaluation methods, however, overlook an essential component: uncertainty, which is crucial for a comprehensive assessment of VLMs. Addressing this oversight, we present a benchmark incorporating uncertainty quantification into evaluating VLMs.
Our analysis spans 20+ VLMs, focusing on the multiple-choice Visual Question Answering (VQA) task. We examine models on 5 datasets that evaluate various …

abstract arxiv assessment benchmark cs.ai cs.cv current evaluation gpt gpt-4 language language models llava oversight performance quantification tasks type uncertainty vision vision-language models vlms