Quantized GAN for Complex Music Generation from Dance Videos. (arXiv:2204.00604v2 [cs.CV] UPDATED)

We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal
framework that generates complex musical samples conditioned on dance videos.
Our proposed framework takes dance video frames and human body motions as
input, and learns to generate music samples that plausibly accompany the
corresponding input. Unlike most existing conditional music generation works
that generate specific types of mono-instrumental sounds using symbolic audio
representations (e.g., MIDI), and that usually rely on pre-defined musical
synthesizers, in this work we generate dance music in complex …

arxiv cv gan generation music videos