Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models

arXiv:2403.16999v1 Announce Type: new
Abstract: This paper presents Visual CoT, a novel pipeline that leverages the reasoning capabilities of multi-modal large language models (MLLMs) by incorporating visual Chain-of-Thought (CoT) reasoning. While MLLMs have shown promise in various visual tasks, they often lack interpretability and struggle with complex visual inputs. To address these challenges, we propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts. We collect and introduce the Visual CoT dataset comprising 373k question-answer …

abstract arxiv capabilities cs.cv inputs interpretability language language models large language large language models mllms modal multi-modal novel paper pipeline reasoning struggle tasks thought type visual