ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

arXiv:2312.00784v2 Announce Type: replace-cross
Abstract: While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a "red bounding box" …

abstract arxiv challenge cs.ai cs.cl cs.cv cs.lg current focus gap image language large multimodal models llava making multimodal multimodal models prompting prompts spatial textual type understanding vision vision-language visual visual prompting while