AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation. (arXiv:2207.10141v1 [cs.SD])

We introduce AudioScopeV2, a state-of-the-art universal audio-visual
on-screen sound separation system which is capable of learning to separate
sounds and associate them with on-screen objects by looking at in-the-wild
videos. We identify several limitations of previous work on audio-visual
on-screen sound separation, including the coarse resolution of spatio-temporal
attention, poor convergence of the audio separation model, limited variety in
training and evaluation data, and failure to account for the trade off between
preservation of on-screen sounds and suppression of off-screen …

arxiv attention audio sound visual attention