Finding Replicable Human Evaluations via Stable Ranking Probability

arXiv:2404.01474v1 Announce Type: new
Abstract: Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial requirement when ranking systems by quality: consistent ranking of systems across repeated evaluations is not just desirable, but essential. Without it, there is no reliable foundation for hill-climbing or product launch decisions. In this paper, we use machine translation and its state-of-the-art human evaluation framework, MQM, as a case study to …

abstract arxiv consistent cs.cl development evaluation human language language generation natural natural language natural language generation probability quality ranking stability systems type via