Dynamic Human Evaluation for Relative Model Comparisons. (arXiv:2112.08048v2 [cs.CL] UPDATED)

Collecting human judgements is currently the most reliable evaluation method
for natural language generation systems. Automatic metrics have reported flaws
when applied to measure quality aspects of generated text and have been shown
to correlate poorly with human judgements. However, human evaluation is time
and cost-intensive, and we lack consensus on designing and conducting human
evaluation experiments. Thus there is a need for streamlined approaches for
efficient collection of human judgements when evaluating natural language
generation systems. Therefore, we present …

arxiv evaluation human