all AI news
Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET. (arXiv:2202.05148v2 [cs.CL] UPDATED)
cs.CL updates on arXiv.org arxiv.org
Neural metrics have achieved impressive correlation with human judgements in
the evaluation of machine translation systems, but before we can safely
optimise towards such metrics, we should be aware of (and ideally eliminate)
biases toward bad translations that receive high scores. Our experiments show
that sample-based Minimum Bayes Risk decoding can be used to explore and
quantify such weaknesses. When applying this strategy to COMET for en-de and
de-en, we find that COMET models are not sensitive enough to discrepancies …
arxiv bayes case case study comet machine machine translation metrics risk study translation