Test for non-negligible adverse shifts. (arXiv:2107.02990v4 [stat.ML] UPDATED)

Statistical tests for dataset shift are susceptible to false alarms: they are
sensitive to minor differences when there is in fact adequate sample coverage
and predictive performance. We propose instead a framework to detect adverse
dataset shifts based on outlier scores, $\texttt{D-SOS}$ for short.
$\texttt{D-SOS}$ holds that the new (test) sample is not substantively worse
than the reference (training) sample, and not that the two are equal. The key
idea is to reduce observations to outlier scores and compare contamination …

arxiv ml test