EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning

arXiv:2403.09502v1 Announce Type: new
Abstract: Recent advancements in self-supervised audio-visual representation learning have demonstrated its potential to capture rich and comprehensive representations. However, despite the advantages of data augmentation verified in many learning methods, audio-visual learning has struggled to fully harness these benefits, as augmentations can easily disrupt the correspondence between input pairs. To address this limitation, we introduce EquiAV, a novel framework that leverages equivariance for audio-visual contrastive learning. Our approach begins with extending equivariance to audio-visual learning, facilitated …

abstract advantages arxiv audio augmentation benefits cs.ai cs.lg data disrupt harness however representation representation learning type visual