Text-to-Audio Generation Synchronized with Videos

arXiv:2403.07938v1 Announce Type: cross
Abstract: In recent times, the focus on text-to-audio (TTA) generation has intensified, as researchers strive to synthesize audio from textual descriptions. However, most existing methods, though leveraging latent diffusion models to learn the correlation between audio and text embeddings, fall short when it comes to maintaining a seamless synchronization between the produced audio and its video. This often results in discernible audio-visual mismatches. To bridge this gap, we introduce a groundbreaking benchmark for Text-to-Audio generation that …

abstract arxiv audio audio generation correlation cs.ai cs.cv cs.lg cs.mm cs.sd diffusion diffusion models eess.as embeddings focus however latent diffusion models learn researchers synchronization text textual type videos