Step fusion: Local and global mutual guidance

arXiv:2306.16950v2 Announce Type: replace
Abstract: Feature alignment is the primary means of fusing multimodal data. We propose a feature alignment method that fully fuses multimodal information, which stepwise shifts and expands feature information from different modalities to have a consistent representation in a feature space. The proposed method can robustly capture high-level interactions between features of different modalities, thus significantly improving the performance of multimodal learning. We also show that the proposed method outperforms other popular multimodal schemes on multiple …

abstract alignment arxiv consistent cs.ai cs.cv data feature fusion global guidance information interactions multimodal multimodal data replace representation space type