Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videos. (arXiv:2206.04523v1 [cs.CL])

In this paper, we propose a neural end-to-end system for voice preserving,
lip-synchronous translation of videos. The system is designed to combine
multiple component models and produces a video of the original speaker speaking
in the target language that is lip-synchronous with the target speech, yet
maintains emphases in speech, voice characteristics, face video of the original
speaker. The pipeline starts with automatic speech recognition including
emphasis detection, followed by a translation model. The translated text is
then synthesized by …

arxiv dubbing face translation videos voice