Unsupervised Audio-Visual Segmentation with Modality Alignment

arXiv:2403.14203v1 Announce Type: new
Abstract: Audio-Visual Segmentation (AVS) aims to identify, at the pixel level, the object in a visual scene that produces a given sound. Current AVS methods rely on costly fine-grained annotations of mask-audio pairs, making them impractical for scalability. To address this, we introduce unsupervised AVS, eliminating the need for such expensive annotation. To tackle this more challenging problem, we propose an unsupervised learning method, named Modality Correspondence Alignment (MoCA), which seamlessly integrates off-the-shelf foundation models like …

abstract alignment annotations arxiv audio avs cs.ai cs.cv current fine-grained identify making object pixel scalability segmentation sound them type unsupervised visual