What if...?: Counterfactual Inception to Mitigate Hallucination Effects in Large Multimodal Models

arXiv:2403.13513v1 Announce Type: new
Abstract: This paper presents a way of enhancing the reliability of Large Multimodal Models (LMMs) in addressing hallucination effects, where models generate incorrect or unrelated responses. Without additional instruction tuning paradigm, we introduce Counterfactual Inception, a novel method that implants counterfactual thoughts into LMMs using carefully chosen, misaligned counterfactual keywords. This method is grounded in the concept of counterfactual thinking, a cognitive process where humans consider alternative realities and outcomes. By applying this human-like reasoning mechanism …

abstract arxiv counterfactual cs.ai cs.cl cs.cv effects generate hallucination large multimodal models lmms multimodal multimodal models novel paper paradigm reliability responses thoughts type