Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking

arXiv:2403.11123v1 Announce Type: new
Abstract: Current metrics for evaluating Dialogue State Tracking (DST) systems exhibit three primary limitations. They: i) erroneously presume a uniform distribution of slots throughout the dialog, ii) neglect to assign partial scores for individual turns, iii) frequently overestimate or underestimate performance by repeatedly counting the models' successful or failed predictions. To address these shortcomings, we introduce a novel metric: Granular Change Accuracy (GCA). GCA focuses on evaluating the predicted changes in dialogue state over the entire …

abstract accuracy arxiv change cs.cl current dialog dialogue distribution iii limitations metrics performance state systems tracking type underestimate uniform