Siamese Vision Transformers are Scalable Audio-visual Learners

arXiv:2403.19638v1 Announce Type: new
Abstract: Traditional audio-visual methods rely on independent audio and visual backbones, which is costly and not scalable. In this work, we investigate using an audio-visual siamese network (AVSiam) for efficient and scalable audio-visual pretraining. Our framework uses a single shared vision transformer backbone to process audio and visual inputs, improving its parameter efficiency, reducing the GPU memory footprint, and allowing us to scale our method to larger datasets and model sizes. We pretrain our model using …

arxiv audio cs.cv cs.sd eess.as scalable transformers type vision vision transformers visual