Leveraging Unlabeled Data to Predict Out-of-Distribution Performance. (arXiv:2201.04234v1 [cs.LG])

Real-world machine learning deployments are characterized by mismatches
between the source (training) and target (test) distributions that may cause
performance drops. In this work, we investigate methods for predicting the
target domain accuracy using only labeled source data and unlabeled target
data. We propose Average Thresholded Confidence (ATC), a practical method that
learns a threshold on the model's confidence, predicting accuracy as the
fraction of unlabeled examples for which model confidence exceeds that
threshold. ATC outperforms previous methods across several …

arxiv data distribution performance