Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents. (arXiv:2201.04723v1 [cs.CL])

At the heart of improving conversational AI is the open problem of how to
evaluate conversations. Issues with automatic metrics are well known (Liu et
al., 2016, arXiv:1603.08023), with human evaluations still considered the gold
standard. Unfortunately, how to perform human evaluations is also an open
problem: differing data collection methods have varying levels of human
agreement and statistical sensitivity, resulting in differing amounts of human
annotation hours and labor costs. In this work we compare five different
crowdworker-based …

agents arxiv human