Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks

arXiv:2404.08347v1 Announce Type: cross
Abstract: Multi-modal learning aims to enhance performance by unifying models from various modalities but often faces the "modality imbalance" problem in real data, leading to a bias towards dominant modalities and neglecting others, thereby limiting its overall effectiveness. To address this challenge, the core idea is to balance the optimization of each modality to achieve a joint optimum. Existing approaches often employ a modal-level control mechanism for adjusting the update of each modal parameter. However, such …

abstract arxiv bias challenge core cs.cv cs.lg data masking modal multi-modal optimization performance real data type