Enhancing Multimodal Cooperation via Fine-grained Modality Valuation

arXiv:2309.06255v3 Announce Type: replace-cross
Abstract: One primary topic of multimodal learning is to jointly incorporate heterogeneous information from different modalities. However, most models often suffer from unsatisfactory multimodal cooperation, which cannot jointly utilize all modalities well. Some methods are proposed to identify and enhance the worse learnt modality, but they are often hard to provide the fine-grained observation of multimodal cooperation at sample-level with theoretical support. Hence, it is essential to reasonably observe and improve the fine-grained cooperation between modalities, …

arxiv cs.ai cs.cv cs.lg cs.mm fine-grained multimodal type valuation via