MoVA: Adapting Mixture of Vision Experts to Multimodal Context

arXiv:2404.13046v1 Announce Type: new
Abstract: As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e.g., the CLIP vision encoder leads to outstanding results on general image understanding but poor performance …

arxiv context cs.cv experts multimodal type vision