Improving Neural Machine Translation by Denoising Training. (arXiv:2201.07365v1 [cs.CL])

We present a simple and effective pretraining strategy {D}en{o}ising
{T}raining DoT for neural machine translation. Specifically, we update the
model parameters with source- and target-side denoising tasks at the early
stage and then tune the model normally. Notably, our approach does not increase
any parameters or training steps, requiring the parallel data merely.
Experiments show that DoT consistently improves the neural machine translation
performance across 12 bilingual and 16 multilingual directions (data size
ranges from 80K to 20M). In addition, …

arxiv machine machine translation neural machine translation training translation