Understanding Generalization via Leave-One-Out Conditional Mutual Information. (arXiv:2206.14800v1 [cs.LG])

We study the mutual information between (certain summaries of) the output of
a learning algorithm and its $n$ training data, conditional on a supersample of
$n+1$ i.i.d. data from which the training data is chosen at random without
replacement. These leave-one-out variants of the conditional mutual information
(CMI) of an algorithm (Steinke and Zakynthinou, 2020) are also seen to control
the mean generalization error of learning algorithms with bounded loss
functions. For learning algorithms achieving zero empirical risk under 0-1 …

arxiv information leave-one-out lg understanding