Unsolvable Problem Detection: Evaluating Trustworthiness of Vision Language Models

arXiv:2403.20331v1 Announce Type: cross
Abstract: This paper introduces a novel and significant challenge for Vision Language Models (VLMs), termed Unsolvable Problem Detection (UPD). UPD examines the VLM's ability to withhold answers when faced with unsolvable problems in the context of Visual Question Answering (VQA) tasks. UPD encompasses three distinct settings: Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Visual Question Detection (IVQD). To deeply investigate the UPD problem, extensive experiments indicate that most VLMs, including GPT-4V and …

abstract arxiv challenge context cs.ai cs.cl cs.cv cs.lg detection language language models novel paper question question answering tasks type vision visual vlm vlms vqa