EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

arXiv:2308.11971v2 Announce Type: replace-cross
Abstract: Building scalable vision-language models to learn from diverse, multimodal data remains an open challenge. In this paper, we introduce an Efficient Vision-languagE foundation model, namely EVE, which is one unified multimodal Transformer pre-trained solely by one unified pre-training task. Specifically, EVE encodes both vision and language within a shared Transformer network integrated with modality-aware sparse Mixture-of-Experts (MoE) modules, which capture modality-specific information by selectively switching to different experts. To unify pre-training tasks of vision and …

abstract arxiv building challenge cs.cl cs.cv cs.lg cs.mm data diverse eve foundation foundation model language language models learn moe multimodal multimodal data paper prediction pre-training scalable training transformer type vision vision-language models