SubER: A Metric for Automatic Evaluation of Subtitle Quality. (arXiv:2205.05805v1 [cs.CL])

This paper addresses the problem of evaluating the quality of automatically
generated subtitles, which includes not only the quality of the
machine-transcribed or translated speech, but also the quality of line
segmentation and subtitle timing. We propose SubER - a single novel metric
based on edit distance with shifts that takes all of these subtitle properties
into account. We compare it to existing metrics for evaluating transcription,
translation, and subtitle quality. A careful human evaluation in a post-editing
scenario shows …

arxiv evaluation quality