Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?

arXiv:2404.18624v1 Announce Type: cross
Abstract: Vision and language models (VLMs) are currently the most generally performant architectures on multimodal tasks. Next to their predictions, they can also produce explanations, either in post-hoc or CoT settings. However, it is not clear how much they use the vision and text modalities when generating predictions or explanations. In this work, we investigate if VLMs rely on modalities differently when generating explanations as opposed to when they provide answers. We also evaluate the self-consistency …

abstract architectures arxiv clear consistent cs.ai cs.cl cs.cv cs.lg however images language language models multimodal next predictions tasks text type vision vlms