CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

arXiv:2405.05949v1 Announce Type: new
Abstract: Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance of improving model capabilities from the vision side. Inspired by the successful applications of Mixture-of-Experts (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo. …

arxiv cs.cv experts llm multimodal scaling type