Is Reference Necessary in the Evaluation of NLG Systems? When and Where?

arXiv:2403.14275v1 Announce Type: new
Abstract: The majority of automatic metrics for evaluating NLG systems are reference-based. However, the challenge of collecting human annotation results in a lack of reliable references in numerous application scenarios. Despite recent advancements in reference-free metrics, it has not been well understood when and where they can be used as an alternative to reference-based metrics. In this study, by employing diverse analytical approaches, we comprehensively assess the performance of both metrics across a wide range of …

abstract annotation application arxiv challenge cs.cl evaluation free however human metrics nlg reference results systems type