A Consistent and Differentiable Lp Canonical Calibration Error Estimator. (arXiv:2210.07810v1 [stat.ML])

Calibrated probabilistic classifiers are models whose predicted probabilities
can directly be interpreted as uncertainty estimates. It has been shown
recently that deep neural networks are poorly calibrated and tend to output
overconfident predictions. As a remedy, we propose a low-bias, trainable
calibration error estimator based on Dirichlet kernel density estimates, which
asymptotically converges to the true $L_p$ calibration error. This novel
estimator enables us to tackle the strongest notion of multiclass calibration,
called canonical (or distribution) calibration, while other common …

arxiv canonical consistent error