StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing

arXiv:2402.12636v1 Announce Type: new
Abstract: Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing state-of-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose …

abstract art arxiv audio challenge cloning cs.cl dubbing emotion generate movie movıe reference scale speech state style type video visual voice voice cloning