Probing Multimodal Large Language Models for Global and Local Semantic Representation

arXiv:2402.17304v1 Announce Type: new
Abstract: The success of large language models has inspired researchers to transfer their exceptional representing ability to other modalities. Several recent works leverage image-caption alignment datasets to train multimodal large language models (MLLMs), which achieve state-of-the-art performance on image-to-text tasks. However, there are very few studies exploring whether MLLMs truly understand the complete image information, i.e., global information, or if they can only capture some local object information. In this study, we find that the intermediate …

abstract alignment art arxiv cs.ai cs.cl datasets global image image-to-text language language models large language large language models mllms multimodal performance representation researchers semantic state success tasks text train transfer type