Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement. (arXiv:2202.04528v3 [cs.SD] UPDATED)

This paper proposes a novel multimodal self-supervised architecture for
energy-efficient audio-visual (AV) speech enhancement that integrates Graph
Neural Networks with canonical correlation analysis (CCA-GNN). The proposed
approach lays its foundations on a state-of-the-art CCA-GNN that learns
representative embeddings by maximizing the correlation between pairs of
augmented views of the same input while decorrelating disconnected features.
The key idea of the conventional CCA-GNN involves discarding
augmentation-variant information and preserving augmentation-invariant
information while preventing capturing of redundant information. Our proposed
AV CCA-GNN …

arxiv audio canonical energy fusion graph graph neural network information multimodal network neural network speech