Octavius: Mitigating Task Interference in MLLMs via MoE

arXiv:2311.02684v1 Announce Type: cross
Abstract: Recent studies have demonstrated Large Language Models (LLMs) can extend their zero-shot generalization capabilities to multimodal learning through instruction tuning. As more modalities and downstream tasks are introduced, negative conflicts and interference may have a worse impact on performance. While this phenomenon has been overlooked in previous work, we propose a novel and extensible framework, called \mname, for comprehensive studies and experimentation on multimodal learning with Multimodal Large Language Models (MLLMs). Specifically, we combine the …

abstract arxiv capabilities cs.cl cs.cv impact interference language language models large language large language models llms mllms moe multimodal multimodal learning negative performance studies tasks through type via zero-shot