Joint Multimodal Transformer for Dimensional Emotional Recognition in the Wild

arXiv:2403.10488v1 Announce Type: cross
Abstract: Audiovisual emotion recognition (ER) in videos has immense potential over unimodal performance. It effectively leverages the inter- and intra-modal dependencies between visual and auditory modalities. This work proposes a novel audio-visual emotion recognition system utilizing a joint multimodal transformer architecture with key-based cross-attention. This framework aims to exploit the complementary nature of audio and visual cues (facial expressions and vocal patterns) in videos, leading to superior performance compared to solely relying on a single modality. …

abstract architecture arxiv attention audio cs.cv cs.lg cs.sd dependencies eess.as emotion framework key modal multimodal novel performance recognition transformer transformer architecture type videos visual work