Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training

arXiv:2404.14604v1 Announce Type: new
Abstract: Open-source multimodal large language models (MLLMs) excel in various tasks involving textual and visual inputs but still struggle with complex multimodal mathematical reasoning, lagging behind proprietary models like GPT-4V(ision) and Gemini-Pro. Although fine-tuning with intermediate steps (i.e., rationales) elicits some mathematical reasoning skills, the resulting models still fall short in visual comprehension due to inadequate visual-centric supervision, which leads to inaccurate interpretation of math figures. To address this issue, we propose a two-step training pipeline …

abstract arxiv cs.cl excel fine-tuning gemini gpt gpt-4v improving inputs intermediate language language models large language large language models mathematical reasoning mllms multimodal proprietary proprietary models reason reasoning skills struggle tasks textual through training type visual