Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision

arXiv:2311.07362v3 Announce Type: replace-cross
Abstract: Large multimodal models suffer from multimodal hallucination, where they provide incorrect responses misaligned with the given visual information. Recent works have conjectured that one of the reasons behind multimodal hallucination is due to the vision encoder failing to ground on the image properly. To mitigate this issue, we propose a novel approach that leverages self-feedback as visual cues. Building on this approach, we introduce Volcano, a multimodal self-feedback guided revision model. Volcano generates natural language …

arxiv cs.cl cs.cv feedback hallucination multimodal through type volcano