Synthesizing Audio from Silent Video using Sequence to Sequence Modeling

arXiv:2404.17608v1 Announce Type: cross
Abstract: Generating audio from a video's visual context has multiple practical applications in improving how we interact with audio-visual media - for example, enhancing CCTV footage analysis, restoring historical videos (e.g., silent movies), and improving video generation models. We propose a novel method to generate audio from video using a sequence-to-sequence model, improving on prior work that used CNNs and WaveNet and faced sound diversity and generalization challenges. Our approach employs a 3D Vector Quantized Variational …

abstract analysis applications arxiv audio cctv context cs.ai cs.lg cs.sd eess.as example generate improving media modeling movies multiple novel practical sequence to sequence type video video generation videos visual