Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generation

arXiv:2305.19556v3 Announce Type: replace
Abstract: Talking face generation is the challenging task of synthesizing a natural and realistic face that requires accurate synchronization with a given audio. Due to co-articulation, where an isolated phone is influenced by the preceding or following phones, the articulation of a phone varies upon the phonetic context. Therefore, modeling lip motion with the phonetic context can generate more spatio-temporally aligned lip movement. In this respect, we investigate the phonetic context in generating lip motion for …

abstract arxiv audio context cs.ai cs.cv cs.sd eess.as eess.iv face natural phone phones synchronization type