Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models

arXiv:2404.04514v1 Announce Type: new
Abstract: Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA), particularly in object-oriented perception tasks which demand fine-grained understanding of object identities, locations or attributes, as indicated by empirical findings. This is mainly due to their limited capability to effectively integrate complex visual cues with textual information and potential object hallucinations. In this paper, we present a novel approach, Joint Visual and Text …

abstract arxiv challenges cs.cl demand face fine-grained gemini gemini pro gpt gpt-4v human language language models large language large language models locations mllms multimodal object object-oriented perception prompting question question answering tasks text type understanding visual vqa