Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning

arXiv:2402.15610v1 Announce Type: new
Abstract: Prior work on selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain. However, when deploying a vision-language system with low tolerance for inaccurate predictions, selective prediction may be over-cautious and abstain too frequently, even on many correct predictions. We introduce ReCoVERR, an inference-time algorithm to reduce the over-abstention of a selective vision-language system without decreasing prediction accuracy. When the VLM makes a low-confidence prediction, instead of …

abstract arxiv cs.cl language language models low prediction predictions prior reasoning selective prediction them type uncertain vision vision-language models vlms work