Data-driven Error Estimation: Upper Bounding Multiple Errors with No Technical Debt

arXiv:2405.04636v1 Announce Type: new
Abstract: We formulate the problem of constructing multiple simultaneously valid confidence intervals (CIs) as estimating a high probability upper bound on the maximum error for a class/set of estimate-estimand-error tuples, and refer to this as the error estimation problem. For a single such tuple, data-driven confidence intervals can often be used to bound the error in our estimate. However, for a class of estimate-estimand-error tuples, nontrivial high probability upper bounds on the maximum error often require …

abstract arxiv class confidence cs.lg data data-driven debt error errors maximum multiple probability set stat.ml technical tuples type