HallE-Control: Controlling Object Hallucination in Large Multimodal Models

arXiv:2310.01779v3 Announce Type: replace
Abstract: Current Large Multimodal Models (LMMs) achieve remarkable progress, yet there remains significant uncertainty regarding their ability to accurately apprehend visual details, that is, in performing detailed captioning. To address this, we introduce $\textit{CCEval}$, a GPT-4 assisted evaluation method for detailed captioning. Interestingly, while LMMs demonstrate minimal object existence hallucination in existing VQA benchmarks, our proposed evaluation reveals continued susceptibility to such hallucinations. In this paper, we make the first attempt to investigate such hallucination from …

abstract arxiv captioning control cs.cv current evaluation gpt gpt-4 hallucination large multimodal models lmms multimodal multimodal models object progress type uncertainty visual