List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs

arXiv:2404.16375v1 Announce Type: new
Abstract: Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of GPT-4V, by enabling the model to associate visual objects with tags inserted on the image. These tags, marked with alphanumerics, can be indexed via text tokens for easy reference. Despite the extraordinary performance from GPT-4V, we observe that other Multimodal Large Language Models (MLLMs) struggle to understand these visual tags. To promote the learning of SoM prompting for open-source models, we propose a new learning paradigm: …

arxiv cs.ai cs.cl cs.cv data list llms multimodal multimodal llms paradigm type