Human-Aligned Calibration for AI-Assisted Decision Making

arXiv:2306.00074v4 Announce Type: replace
Abstract: Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of …

abstract arxiv binary classifier confidence context cs.cy cs.hc cs.lg decision decision making decision support human maker making prediction stat.ml support trust type value