Empirical Comparison between Cross-Validation and Mutation-Validation in Model Selection

arXiv:2311.14079v2 Announce Type: replace
Abstract: Mutation validation (MV) is a recently proposed approach for model selection, garnering significant interest due to its unique characteristics and potential benefits compared to the widely used cross-validation (CV) method. In this study, we empirically compared MV and $k$-fold CV using benchmark and real-world datasets. By employing Bayesian tests, we compared generalization estimates yielding three posterior probabilities: practical equivalence, CV superiority, and MV superiority. We also evaluated the differences in the capacity of the selected …

abstract arxiv benchmark benefits comparison cs.lg datasets model selection mutation stat.ml study type validation world