Sparse multimodal fusion with modal channel attention

arXiv:2403.20280v1 Announce Type: new
Abstract: The ability of masked multimodal transformer architectures to learn a robust embedding space when modality samples are sparsely aligned is studied by measuring the quality of generated embedding spaces as a function of modal sparsity. An extension to the masked multimodal transformer model is proposed which incorporates modal-incomplete channels in the multihead attention mechanism called modal channel attention (MCA). Two datasets with 4 modalities are used, CMU-MOSEI for multimodal sentiment recognition and TCGA for multiomics. …

abstract architectures arxiv attention cs.ai cs.lg embedding extension function fusion generated learn measuring modal multimodal quality robust samples space spaces sparsity transformer transformer model type