Can we globally optimize cross-validation loss? Quasiconvexity in ridge regression. (arXiv:2107.09194v2 [stat.ML] UPDATED)

Models like LASSO and ridge regression are extensively used in practice due
to their interpretability, ease of use, and strong theoretical guarantees.
Cross-validation (CV) is widely used for hyperparameter tuning in these models,
but do practical optimization methods minimize the true out-of-sample loss? A
recent line of research promises to show that the optimum of the CV loss
matches the optimum of the out-of-sample loss (possibly after simple
corrections). It remains to show how tractable it is to minimize the …

arxiv loss regression ridge validation validation loss