Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective

arXiv:2403.18346v1 Announce Type: cross
Abstract: Recent advancements in Large Language Models (LLMs) have facilitated the development of Multimodal LLMs (MLLMs). Despite their impressive capabilities, MLLMs often suffer from an over-reliance on unimodal biases (e.g., language bias and vision bias), leading to incorrect answers in complex multimodal tasks. To investigate this issue, we propose a causal framework to interpret the biases in Visual Question Answering (VQA) problems. Within our framework, we devise a causal graph to elucidate the predictions of MLLMs …

abstract arxiv bias biases capabilities causal cs.cl cs.cv development language language models large language large language models llms mllms multimodal multimodal llms perspective reliance tasks type vision