Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners

arXiv:2402.17723v1 Announce Type: new
Abstract: Video and audio content creation serves as the core technique for the movie industry and professional users. Recently, existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation. We observe the powerful generation ability of off-the-shelf video or audio generation models. Thus, instead of training the giant …

arxiv audio audio generation cs.cv diffusion domain hearing type visual