Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation. (arXiv:2308.03131v3 [cs.CL] UPDATED)

N-gram matching-based evaluation metrics, such as BLEU and chrF, are widely
utilized across a range of natural language generation (NLG) tasks. However,
recent studies have revealed a weak correlation between these matching-based
metrics and human evaluations, especially when compared with neural-based
metrics like BLEURT. In this paper, we conjecture that the performance
bottleneck in matching-based metrics may be caused by the limited diversity of
references. To address this issue, we propose to utilize \textit{multiple
references} to enhance the consistency between …

arxiv correlation data data leakage diversity evaluation evaluation metrics human language language generation metrics multiple natural natural language natural language generation nlg reference studies