Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Models

arXiv:2403.16167v1 Announce Type: new
Abstract: Hallucinations in vision-language models pose a significant challenge to their reliability, particularly in the generation of long captions. Current methods fall short of accurately identifying and mitigating these hallucinations. To address this issue, we introduce ESREAL, a novel unsupervised learning framework designed to suppress the generation of hallucinations through accurate localization and penalization of hallucinated tokens. Initially, ESREAL creates a reconstructed image based on the generated caption and aligns its corresponding regions with those of …

abstract arxiv captions challenge cs.cl cs.cv current framework hallucinations issue language language models novel reliability semantic type unsupervised unsupervised learning vision vision-language models