Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationales

arXiv:2404.11129v1 Announce Type: new
Abstract: The remarkable performance of Multimodal Large Language Models (MLLMs) has unequivocally demonstrated their proficient understanding capabilities in handling a wide array of visual tasks. Nevertheless, the opaque nature of their black-box reasoning processes persists as an enigma, rendering them uninterpretable and struggling with hallucination. Their ability to execute intricate compositional reasoning tasks is also constrained, culminating in a stagnation of learning progression for these models. In this work, we introduce Fact, a novel paradigm designed …

abstract array arxiv box capabilities cs.cv hallucination language language models large language large language models mllms multimodal nature performance processes reasoning rendering tasks teaching them type understanding visual