Multiplicity-adjusted bootstrap tilting lower confidence bounds for conditional prediction performance measures. (arXiv:2210.13206v1 [stat.ML])

In machine learning, the selection of a promising model from a potentially
large number of competing models and the assessment of its generalization
performance are critical tasks that need careful consideration. Typically,
model selection and evaluation are strictly separated endeavors, splitting the
sample at hand into a training, validation, and evaluation set, and only
compute a single confidence interval for the prediction performance of the
final selected model. We however propose an algorithm how to compute valid
lower confidence bounds …

arxiv bootstrap confidence performance prediction