Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach

arXiv:2401.10747v2 Announce Type: replace-cross
Abstract: Multimodal sentiment analysis aims to identify the emotions expressed by individuals through visual, language, and acoustic cues. However, most of the existing research efforts assume that all modalities are available during both training and testing, making their algorithms susceptible to the missing modality scenario. In this paper, we propose a novel knowledge-transfer network to translate between different modalities to reconstruct the missing audio modalities. Moreover, we develop a cross-modality attention mechanism to retain the maximal …

abstract algorithms analysis arxiv cs.ai cs.cl cs.lg cs.sd eess.as emotions however identify knowledge language making multimodal replace research sentiment sentiment analysis testing through training transfer type visual