Hallucination Augmented Contrastive Learning for Multimodal Large Language Model

arXiv:2312.06968v4 Announce Type: replace
Abstract: Multi-modal large language models (MLLMs) have been shown to efficiently integrate natural language with visual information to handle multi-modal tasks. However, MLLMs still face a fundamental limitation of hallucinations, where they tend to generate erroneous or fabricated information. In this paper, we address hallucinations in MLLMs from a novel perspective of representation learning. We first analyzed the representation distribution of textual and visual tokens in MLLM, revealing two important findings: 1) there is a significant …

arxiv cs.cv hallucination language language model large language large language model multimodal multimodal large language model type