On Leave-One-Out Conditional Mutual Information For Generalization. (arXiv:2207.00581v1 [cs.LG])

We derive information theoretic generalization bounds for supervised learning
algorithms based on a new measure of leave-one-out conditional mutual
information (loo-CMI). Contrary to other CMI bounds, which are black-box bounds
that do not exploit the structure of the problem and may be hard to evaluate in
practice, our loo-CMI bounds can be computed easily and can be interpreted in
connection to other notions such as classical leave-one-out cross-validation,
stability of the optimization algorithm, and the geometry of the
loss-landscape. It …

arxiv information leave-one-out lg