TinyLLaVA: A Framework of Small-scale Large Multimodal Models

arXiv:2402.14289v1 Announce Type: new
Abstract: We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects of different vision encoders, connection modules, language models, training data and training recipes. Our extensive experiments showed that better quality of data combined with better training recipes, smaller LMMs can consistently achieve on-par performances compared to bigger LMMs. Under our framework, we train a family of small-scale LMMs. Our best …

abstract arxiv cs.cl cs.lg data designing effects framework language language models large multimodal models lmms modules multimodal multimodal models perspective quality recipes scale small study training training data type vision