SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis

arXiv:2311.17590v2 Announce Type: replace
Abstract: Achieving high synchronization in the synthesis of realistic, speech-driven talking head videos presents a significant challenge. Traditional Generative Adversarial Networks (GAN) struggle to maintain consistent facial identity, while Neural Radiance Fields (NeRF) methods, although they can address this issue, often produce mismatched lip movements, inadequate facial expressions, and unstable head poses. A lifelike talking head requires synchronized coordination of subject identity, lip movements, facial expressions, and head poses. The absence of these synchronizations is a …

arxiv cs.cv head synchronization synthesis talking head type