Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation

arXiv:2312.06462v2 Announce Type: replace
Abstract: Recently, an audio-visual segmentation (AVS) task has been introduced, aiming to group pixels with sounding objects within a given video. This task necessitates a first-ever audio-driven pixel-level understanding of the scene, posing significant challenges. In this paper, we propose an innovative audio-visual transformer framework, termed COMBO, an acronym for COoperation of Multi-order Bilateral relatiOns. For the first time, our framework explores three types of bilateral entanglements within AVS: pixel entanglement, modality entanglement, and temporal entanglement. …

arxiv audio cs.ai cs.cv cs.sd eess.as matter relations segmentation type visual