Online Calibrated and Conformal Prediction Improves Bayesian Optimization

arXiv:2112.04620v4 Announce Type: replace
Abstract: Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.g., Gaussianity). This paper studies which uncertainties are needed in model-based decision-making and in Bayesian optimization, and argues that uncertainties can benefit from calibration -- i.e., an 80% predictive interval should contain the true outcome 80% of the time. Maintaining calibration, however, can be challenging when …

abstract arxiv assumptions bayesian cs.lg data decision however making optimization paper prediction stat.ml studies tasks type uncertainty