Sources of Irreproducibility in Machine Learning: A Review. (arXiv:2204.07610v2 [cs.LG] UPDATED)

Background: Many published machine learning studies are irreproducible.
Issues with methodology and not properly accounting for variation introduced by
the algorithm themselves or their implementations are attributed as the main
contributors to the irreproducibility.Problem: There exist no theoretical
framework that relates experiment design choices to potential effects on the
conclusions. Without such a framework, it is much harder for practitioners and
researchers to evaluate experiment results and describe the limitations of
experiments. The lack of such a framework also makes …

accounting algorithm arxiv contributors design effects experiment framework independent machine machine learning methodology researchers review studies