Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models

arXiv:2403.19322v1 Announce Type: new
Abstract: The surge of Multimodal Large Language Models (MLLMs), given their prominent emergent capabilities in instruction following and reasoning, has greatly advanced the field of visual reasoning. However, constrained by their non-lossless image tokenization, most MLLMs fall short of comprehensively capturing details of text and objects, especially in high-resolution images. To address this, we propose P2G, a novel framework for plug-and-play grounding of reasoning in MLLMs. Specifically, P2G exploits the tool-usage potential of MLLMs to employ …

abstract advanced arxiv capabilities cs.cl cs.cv however image language language models large language large language models mllms multimodal objects reasoning text tokenization type visual