All in a Single Image: Large Multimodal Models are In-Image Learners

arXiv:2402.17971v1 Announce Type: new
Abstract: This paper introduces a new in-context learning (ICL) mechanism called In-Image Learning (I$^2$L) that combines demonstration examples, visual cues, and instructions into a single image to enhance the capabilities of GPT-4V. Unlike previous approaches that rely on converting images to text or incorporating visual input into language models, I$^2$L consolidates all information into one image and primarily leverages image processing, understanding, and reasoning abilities. This has several advantages: it avoids inaccurate textual descriptions of complex …

arxiv cs.ai cs.cl cs.cv image large multimodal models multimodal multimodal models type