IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation

Recent research has made significant progress in designing fusion modules for audio-visual speech separation. However, they predominantly focus on multi-modal fusion at a single temporal scale of auditory and visual features without employing selective attention mechanisms, which is in sharp contrast with the brain. To address this issue, We propose a novel model called Intra- and Inter-Attention Network (IIANet), which leverages the attention mechanism for efficient audio-visual feature fusion. IIANet consists of two types of attention blocks: intra-attention (IntraA) and …

attention attention mechanisms audio brain contrast cs.cv cs.mm cs.sd designing eess.as features focus fusion issue modal modules multi-modal network progress research scale speech temporal visual