The Calibration Generalization Gap. (arXiv:2210.01964v2 [cs.LG] UPDATED)

Calibration is a fundamental property of a good predictive model: it requires
that the model predicts correctly in proportion to its confidence. Modern
neural networks, however, provide no strong guarantees on their calibration --
and can be either poorly calibrated or well-calibrated depending on the
setting. It is currently unclear which factors contribute to good calibration
(architecture, data augmentation, overparameterization, etc), though various
claims exist in the literature.

We propose a systematic way to study the calibration error: by decomposing …

arxiv gap