Rashomon Capacity: A Metric for Predictive Multiplicity in Classification. (arXiv:2206.01295v2 [cs.LG] UPDATED)

Predictive multiplicity occurs when classification models with statistically
indistinguishable performances assign conflicting predictions to individual
samples. When used for decision-making in applications of consequence (e.g.,
lending, education, criminal justice), models developed without regard for
predictive multiplicity may result in unjustified and arbitrary decisions for
specific individuals. We introduce a new metric, called Rashomon Capacity, to
measure predictive multiplicity in probabilistic classification. Prior metrics
for predictive multiplicity focus on classifiers that output thresholded (i.e.,
0-1) predicted classes. In contrast, Rashomon Capacity …

arxiv capacity classification predictive