Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models

arXiv:2310.00840v2 Announce Type: replace
Abstract: Text generation models are notoriously vulnerable to errors in the training data. With the wide-spread availability of massive amounts of web-crawled data becoming more commonplace, how can we enhance the robustness of models trained on a massive amount of noisy web-crawled text? In our work, we propose Error Norm Truncation (ENT), a robust enhancement method to the standard training objective that truncates noisy data. Compared to methods that only uses the negative log-likelihood loss to …

abstract arxiv availability cs.cl data error errors massive noise norm robust robustness text text generation training training data type vulnerable web