Escaping The Curse of Dimensionality in Bayesian Model-Based Clustering

Bayesian mixture models are widely used for clustering of high-dimensional data with appropriate uncertainty quantification. However, as the dimension of the observations increases, posterior inference often tends to favor too many or too few clusters. This article explains this behavior by studying the random partition posterior in a non-standard setting with a fixed sample size and increasing data dimensionality. We provide conditions under which the finite sample posterior tends to either assign every observation to a different cluster or all …

article bayesian behavior clustering data dimensionality inference posterior quantification random standard studying the curse of dimensionality uncertainty