Optimal Ridge Regularization for Out-of-Distribution Prediction

arXiv:2404.01233v1 Announce Type: cross
Abstract: We study the behavior of optimal ridge regularization and optimal ridge risk for out-of-distribution prediction, where the test distribution deviates arbitrarily from the train distribution. We establish general conditions that determine the sign of the optimal regularization level under covariate and regression shifts. These conditions capture the alignment between the covariance and signal structures in the train and test data and reveal stark differences compared to the in-distribution setting. For example, a negative regularization level …

abstract alignment arxiv behavior cs.lg distribution general math.st prediction regression regularization ridge risk stat.ml stat.th study test train type