Deep Clustering Evaluation: How to Validate Internal Clustering Validation Measures

arXiv:2403.14830v1 Announce Type: cross
Abstract: Deep clustering, a method for partitioning complex, high-dimensional data using deep neural networks, presents unique evaluation challenges. Traditional clustering validation measures, designed for low-dimensional spaces, are problematic for deep clustering, which involves projecting data into lower-dimensional embeddings before partitioning. Two key issues are identified: 1) the curse of dimensionality when applying these measures to raw data, and 2) the unreliable comparison of clustering results across different embedding spaces stemming from variations in training procedures and …

abstract arxiv challenges clustering cs.lg data embeddings evaluation key low networks neural networks partitioning spaces stat.ml type validation