Modeling the Machine Learning Multiverse. (arXiv:2206.05985v2 [cs.LG] UPDATED)

Amid mounting concern about the reliability and credibility of machine
learning research, we present a principled framework for making robust and
generalizable claims: the multiverse analysis. Our framework builds upon the
multiverse analysis (Steegen et al., 2016) introduced in response to
psychology's own reproducibility crisis. To efficiently explore
high-dimensional and often continuous ML search spaces, we model the multiverse
with a Gaussian Process surrogate and apply Bayesian experimental design. Our
framework is designed to facilitate drawing robust scientific conclusions about …

arxiv machine machine learning modeling multiverse