VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis

arXiv:2403.08764v1 Announce Type: new
Abstract: We propose VLOGGER, a method for audio-driven human video generation from a single input image of a person, which builds on the success of recent generative diffusion models. Our method consists of 1) a stochastic human-to-3d-motion diffusion model, and 2) a novel diffusion-based architecture that augments text-to-image models with both spatial and temporal controls. This supports the generation of high quality video of variable length, easily controllable through high-level representations of human faces and bodies. …

abstract architecture arxiv audio avatar cs.cv diffusion diffusion model diffusion models embodied generative human image multimodal novel person stochastic success synthesis text text-to-image type video video generation