Multimodal Transformer Distillation for Audio-Visual Synchronization

arXiv:2210.15563v3 Announce Type: replace
Abstract: Audio-visual synchronization aims to determine whether the mouth movements and speech in the video are synchronized. VocaLiST reaches state-of-the-art performance by incorporating multimodal Transformers to model audio-visual interact information. However, it requires high computing resources, making it impractical for real-world applications. This paper proposed an MTDVocaLiST model, which is trained by our proposed multimodal Transformer distillation (MTD) loss. MTD loss enables MTDVocaLiST model to deeply mimic the cross-attention distribution and value-relation in the Transformer of …

abstract applications art arxiv audio computing computing resources cs.cv cs.ir cs.sd distillation eess.as however information making movements multimodal paper performance resources speech state synchronization transformer transformers type video visual world