GAIA: Zero-shot Talking Avatar Generation

arXiv:2311.15230v2 Announce Type: replace
Abstract: Zero-shot talking avatar generation aims at synthesizing natural talking videos from speech and a single portrait image. Previous methods have relied on domain-specific heuristics such as warping-based motion representation and 3D Morphable Models, which limit the naturalness and diversity of the generated avatars. In this work, we introduce GAIA (Generative AI for Avatar), which eliminates the domain priors in talking avatar generation. In light of the observation that the speech only drives the motion of …

abstract arxiv avatar avatars cs.cv cs.mm diversity domain generated generative heuristics image natural representation speech type videos work zero-shot