Reproducibility in Learning. (arXiv:2201.08430v2 [cs.LG] UPDATED)

We introduce the notion of a reproducible algorithm in the context of
learning. A reproducible learning algorithm is resilient to variations in its
samples -- with high probability, it returns the exact same output when run on
two samples from the same underlying distribution. We begin by unpacking the
definition, clarifying how randomness is instrumental in balancing accuracy and
reproducibility. We initiate a theory of reproducible algorithms, showing how
reproducibility implies desirable properties such as data reuse and efficient
testability. …

accuracy algorithm algorithms arxiv context data definition demand distribution notion probability randomness reproducibility resilient returns theory