[R] Maximum likelihood estimation can fail due to "Manifold Overfitting"

arXiv: https://arxiv.org/abs/2204.07172

This paper out today seems to make the bold claim that maximum likelihood estimation is not a well-posed training objective in deep generative modelling. The manifold hypothesis says that observed high-dimensional data clusters around low-dimensional manifolds, but maximum likelihood methods (e.g. VAE, normalizing flows) learn high-dimensional densities. The paper argues that the mismatch between dimensionalities will lead to a problem called "manifold overfitting".

Models are able to maximize likelihood in high-dimensions by sending the density to infinity around …

machinelearning manifold maximum likelihood estimation overfitting