Hierarchical clustering with dot products recovers hidden tree structure

arXiv:2305.15022v3 Announce Type: replace-cross
Abstract: In this paper we offer a new perspective on the well established agglomerative clustering algorithm, focusing on recovery of hierarchical structure. We recommend a simple variant of the standard algorithm, in which clusters are merged by maximum average dot product and not, for example, by minimum distance or within-cluster variance. We demonstrate that the tree output by this algorithm provides a bona fide estimate of generative hierarchical structure in data, under a generic probabilistic graphical …

abstract algorithm arxiv clustering clustering algorithm cs.lg example hidden hierarchical paper perspective product products recovery simple standard stat.ml tree type