Diagnosing and Fixing Manifold Overfitting in Deep Generative Models. (arXiv:2204.07172v3 [stat.ML] UPDATED)

Likelihood-based, or explicit, deep generative models use neural networks to
construct flexible high-dimensional densities. This formulation directly
contradicts the manifold hypothesis, which states that observed data lies on a
low-dimensional manifold embedded in high-dimensional ambient space. In this
paper we investigate the pathologies of maximum-likelihood training in the
presence of this dimensionality mismatch. We formally prove that degenerate
optima are achieved wherein the manifold itself is learned but not the
distribution on it, a phenomenon we call manifold overfitting. We …

arxiv deep generative models manifold ml overfitting