What is the $\textit{intrinsic}$ dimension of your binary data? -- and how to compute it quickly

arXiv:2404.06326v1 Announce Type: new
Abstract: Dimensionality is an important aspect for analyzing and understanding (high-dimensional) data. In their 2006 ICDM paper Tatti et al. answered the question for a (interpretable) dimension of binary data tables by introducing a normalized correlation dimension. In the present work we revisit their results and contrast them with a concept based notion of intrinsic dimension (ID) recently introduced for geometric data sets. To do this, we present a novel approximation for this ID that is …

abstract arxiv binary compute correlation cs.ai cs.lg data dimensionality intrinsic paper question tables type understanding