Multi-Modal Hallucination Control by Visual Information Grounding

arXiv:2403.14003v1 Announce Type: cross
Abstract: Generative Vision-Language Models (VLMs) are prone to generate plausible-sounding textual answers that, however, are not always grounded in the input image. We investigate this phenomenon, usually referred to as "hallucination" and show that it stems from an excessive reliance on the language prior. In particular, we show that as more tokens are generated, the reliance on the visual prompt decreases, and this behavior strongly correlates with the emergence of hallucinations. To reduce hallucinations, we introduce …

abstract arxiv control cs.cl cs.cv cs.lg generate generative hallucination however image information language language models modal multi-modal prior reliance show textual type vision vision-language models visual vlms