Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos

arXiv:2406.09272v1 Announce Type: new
Abstract: Generating realistic audio for human interactions is important for many applications, such as creating sound effects for films or virtual reality games. Existing approaches implicitly assume total correspondence between the video and audio during training, yet many sounds happen off-screen and have weak to no correspondence with the visuals -- resulting in uncontrolled ambient sounds or hallucinations at test time. We propose a novel ambient-aware audio generation model, AV-LDM. We devise a novel audio-conditioning mechanism …

abstract action ambient applications arxiv audio cs.ai cs.cv cs.sd eess.as effects films games human human interactions important interactions off reality sound sound effects total training type video videos virtual virtual reality virtual reality games