A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice

arXiv:2404.16958v1 Announce Type: new
Abstract: Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Frequently, metrics are selected without arguments, and blurry terminology invites misconceptions. For instance, many works use so-called 'macro' metrics to rank systems (e.g., 'macro F1') but do not clearly specify what they would expect from such a 'macro' metric. This is problematic, since picking a metric can affect paper findings as well as shared task rankings, …

abstract arxiv classification closer look cs.cl cs.lg evaluation evaluation metrics however instance look macro metrics papers practice systems terminology type