MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

arXiv:2403.09611v1 Announce Type: cross
Abstract: In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) …

abstract analysis architecture arxiv building components cs.cl cs.cv cs.lg data discuss encoder image importance insights language language models large language large language models llm mllms multimodal pre-training study through training training data type vision work