Revisiting Meta-evaluation for Grammatical Error Correction

arXiv:2403.02674v1 Announce Type: new
Abstract: Metrics are the foundation for automatic evaluation in grammatical error correction (GEC), with their evaluation of the metrics (meta-evaluation) relying on their correlation with human judgments. However, conventional meta-evaluations in English GEC encounter several challenges including biases caused by inconsistencies in evaluation granularity, and an outdated setup using classical systems. These problems can lead to misinterpretation of metrics and potentially hinder the applicability of GEC techniques. To address these issues, this paper proposes SEEDA, a …

abstract arxiv biases challenges correlation cs.cl english error error correction evaluation foundation gec human meta metrics setup type