Prosodic Alignment for off-screen automatic dubbing. (arXiv:2204.02530v1 [cs.CL])

The goal of automatic dubbing is to perform speech-to-speech translation
while achieving audiovisual coherence. This entails isochrony, i.e.,
translating the original speech by also matching its prosodic structure into
phrases and pauses, especially when the speaker's mouth is visible. In previous
work, we introduced a prosodic alignment model to address isochrone or
on-screen dubbing. In this work, we extend the prosodic alignment model to also
address off-screen dubbing that requires less stringent synchronization
constraints. We conduct experiments on four dubbing …

arxiv dubbing