LDReg: Local Dimensionality Regularized Self-Supervised Learning

arXiv:2401.10474v2 Announce Type: replace
Abstract: Representations learned via self-supervised learning (SSL) can be susceptible to dimensional collapse, where the learned representation subspace is of extremely low dimensionality and thus fails to represent the full data distribution and modalities. Dimensional collapse also known as the "underfilling" phenomenon is one of the major causes of degraded performance on downstream tasks. Previous work has investigated the dimensional collapse problem of SSL at a global level. In this paper, we demonstrate that representations can …

abstract arxiv cs.ai cs.cv cs.lg data dimensionality distribution low major representation self-supervised learning ssl stat.ml supervised learning type via