SALTED: A Framework for SAlient Long-Tail Translation Error Detection. (arXiv:2205.09988v1 [cs.CL])

Traditional machine translation (MT) metrics provide an average measure of
translation quality that is insensitive to the long tail of behavioral problems
in MT. Examples include translation of numbers, physical units, dropped content
and hallucinations. These errors, which occur rarely and unpredictably in
Neural Machine Translation (NMT), greatly undermine the reliability of
state-of-the-art MT systems. Consequently, it is important to have visibility
into these problems during model development. Towards this direction, we
introduce SALTED, a specifications-based framework for behavioral testing …

arxiv detection error framework translation