Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. (arXiv:2110.01052v4 [cs.LG] UPDATED)

We introduce a framework for calibrating machine learning models so that
their predictions satisfy explicit, finite-sample statistical guarantees. Our
calibration algorithm works with any underlying model and (unknown)
data-generating distribution and does not require model refitting. The
framework addresses, among other examples, false discovery rate control in
multi-label classification, intersection-over-union control in instance
segmentation, and the simultaneous control of the type-1 error of outlier
detection and confidence set coverage in classification or regression. Our main
insight is to reframe the …

algorithms arxiv predictive risk test