MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition

arXiv:2404.18327v1 Announce Type: new
Abstract: This paper presents a novel approach to processing multimodal data for dynamic emotion recognition, named as the Multimodal Masked Autoencoder for Dynamic Emotion Recognition (MultiMAE-DER). The MultiMAE-DER leverages the closely correlated representation information within spatiotemporal sequences across visual and audio modalities. By utilizing a pre-trained masked autoencoder model, the MultiMAEDER is accomplished through simple, straightforward finetuning. The performance of the MultiMAE-DER is enhanced by optimizing six fusion strategies for multimodal input sequences. These strategies address …

abstract arxiv audio autoencoder cs.cv data dynamic emotion information masked autoencoder multimodal multimodal data novel paper processing recognition representation type visual