MINDE: Mutual Information Neural Diffusion Estimation

arXiv:2310.09031v2 Announce Type: replace
Abstract: In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to estimate the Kullback Leibler divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables. Armed with such building blocks, we …

abstract arxiv cs.lg diffusion diffusion models divergence information interpretation random replace stat.ml theorem type variables work