On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifiers. (arXiv:2203.08958v3 [cs.LG] UPDATED)

Web: http://arxiv.org/abs/2203.08958

Every uncalibrated classifier has a corresponding true calibration map that
calibrates its confidence. Deviations of this idealistic map from the identity
map reveal miscalibration. Such calibration errors can be reduced with many
post-hoc calibration methods which fit some family of calibration maps on a
validation dataset. In contrast, evaluation of calibration with the expected
calibration error (ECE) on the test set does not explicitly involve fitting.
However, as we demonstrate, ECE can still be viewed as if fitting a family …

arxiv on test