Talk3D: High-Fidelity Talking Portrait Synthesis via Personalized 3D Generative Prior

arXiv:2403.20153v1 Announce Type: new
Abstract: Recent methods for audio-driven talking head synthesis often optimize neural radiance fields (NeRF) on a monocular talking portrait video, leveraging its capability to render high-fidelity and 3D-consistent novel-view frames. However, they often struggle to reconstruct complete face geometry due to the absence of comprehensive 3D information in the input monocular videos. In this paper, we introduce a novel audio-driven talking head synthesis framework, called Talk3D, that can faithfully reconstruct its plausible facial geometries by effectively …

abstract arxiv audio capability consistent cs.cv face fidelity fields generative geometry head however nerf neural radiance fields novel personalized prior struggle synthesis talking head type via video view