AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

arXiv:2403.17694v1 Announce Type: new
Abstract: In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority …

animation arxiv audio cs.cv cs.gr eess.iv photorealistic synthesis type