Testing Calibration in Subquadratic Time

arXiv:2402.13187v1 Announce Type: new
Abstract: In the recent literature on machine learning and decision making, calibration has emerged as a desirable and widely-studied statistical property of the outputs of binary prediction models. However, the algorithmic aspects of measuring model calibration have remained relatively less well-explored. Motivated by [BGHN23], which proposed a rigorous framework for measuring distances to calibration, we initiate the algorithmic study of calibration through the lens of property testing. We define the problem of calibration testing from samples …

abstract arxiv binary cs.ds cs.lg decision decision making framework literature machine machine learning making measuring prediction prediction models property stat.co statistical stat.ml testing type