MoPE: Parameter-Efficient and Scalable Multimodal Fusion via Mixture of Prompt Experts

arXiv:2403.10568v1 Announce Type: new
Abstract: Prompt-tuning has demonstrated parameter-efficiency in fusing unimodal foundation models for multimodal tasks. However, its limited adaptivity and expressiveness lead to suboptimal performance when compared with other tuning methods. In this paper, we address this issue by disentangling the vanilla prompts to adaptively capture dataset-level and instance-level features. Building upon this disentanglement, we introduce the mixture of prompt experts (MoPE) technique to enhance expressiveness. MoPE leverages multimodal pairing priors to route the most effective prompt on …

arxiv cs.ai cs.cl cs.cv cs.lg experts fusion multimodal prompt scalable type via