Reference-based Metrics Disprove Themselves in Question Generation

arXiv:2403.12242v1 Announce Type: cross
Abstract: Reference-based metrics such as BLEU and BERTScore are widely used to evaluate question generation (QG). In this study, on QG benchmarks such as SQuAD and HotpotQA, we find that using human-written references cannot guarantee the effectiveness of the reference-based metrics. Most QG benchmarks have only one reference; we replicated the annotation process and collect another reference. A good metric was expected to grade a human-validated question no worse than generated questions. However, the results of …

abstract arxiv benchmarks bleu cs.ai cs.cl cs.lg human metrics question reference study type