Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning

arXiv:2403.15469v1 Announce Type: cross
Abstract: Traditional Automatic Video Dubbing (AVD) pipeline consists of three key modules, namely, Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS). Within AVD pipelines, isometric-NMT algorithms are employed to regulate the length of the synthesized output text. This is done to guarantee synchronization with respect to the alignment of video and audio subsequent to the dubbing process. Previous approaches have focused on aligning the number of characters and words in the source and …

abstract algorithms arxiv asr automatic speech recognition count cs.cl cs.lg dubbing eess.as key machine machine translation modules neural machine translation pipeline pipelines recognition reinforcement reinforcement learning speech speech recognition synthesized text text-to-speech translation tts type video