Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularization

arXiv:2402.09725v1 Announce Type: new
Abstract: Being one of the IR-NAT (Iterative-refinemennt-based NAT) frameworks, the Conditional Masked Language Model (CMLM) adopts the mask-predict paradigm to re-predict the masked low-confidence tokens. However, CMLM suffers from the data distribution discrepancy between training and inference, where the observed tokens are generated differently in the two cases. In this paper, we address this problem with the training approaches of error exposure and consistency regularization (EECR). We construct the mixed sequences based on model prediction during …

abstract arxiv confidence cs.ai cs.cl data distribution error frameworks generated inference iterative language language model low machine machine translation nat paradigm regularization tokens training translation type