Tracking the risk of a deployed model and detecting harmful distribution shifts. (arXiv:2110.06177v4 [stat.ML] UPDATED)

When deployed in the real world, machine learning models inevitably encounter
changes in the data distribution, and certain -- but not all -- distribution
shifts could result in significant performance degradation. In practice, it may
make sense to ignore benign shifts, under which the performance of a deployed
model does not degrade substantially, making interventions by a human expert
(or model retraining) unnecessary. While several works have developed tests for
distribution shifts, these typically either use non-sequential methods, or
detect …

arxiv distribution ml risk tracking