Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imagination

arXiv:2401.08025v2 Announce Type: replace-cross
Abstract: The potential of Vision-Language Models (VLMs) often remains underutilized in handling complex text-based problems, particularly when these problems could benefit from visual representation. Resonating with humans' ability to solve complex text-based problems by (1) creating a visual diagram from the problem and (2) deducing what steps they need to take to solve it, we propose Self-Imagine. We leverage a single Vision-Language Model (VLM) to generate a structured representation of the question using HTML, then render …

abstract arxiv benefit cs.ai cs.cl cs.lg humans imagination imagine language language models multimodal multimodal models reasoning representation solve text type vision vision-language models visual vlms