MANTIS: Interleaved Multi-Image Instruction Tuning

arXiv:2405.01483v1 Announce Type: cross
Abstract: The recent years have witnessed a great array of large multimodal models (LMMs) to effectively solve single-image vision language tasks. However, their abilities to solve multi-image visual language tasks is yet to be improved. The existing multi-image LMMs (e.g. OpenFlamingo, Emu, Idefics, etc) mostly gain their multi-image ability through pre-training on hundreds of millions of noisy interleaved image-text data from web, which is neither efficient nor effective. In this paper, we aim at building strong …

abstract array arxiv cs.ai cs.cl cs.cv emu etc however image instruction tuning language large multimodal models lmms multimodal multimodal models pre-training solve tasks through training type vision visual