WERank: Towards Rank Degradation Prevention for Self-Supervised Learning Using Weight Regularization

arXiv:2402.09586v1 Announce Type: new
Abstract: A common phenomena confining the representation quality in Self-Supervised Learning (SSL) is dimensional collapse (also known as rank degeneration), where the learned representations are mapped to a low dimensional subspace of the representation space. The State-of-the-Art SSL methods have shown to suffer from dimensional collapse and fall behind maintaining full rank. Recent approaches to prevent this problem have proposed using contrastive losses, regularization techniques, or architectural tricks. We propose WERank, a new regularizer on the …

abstract art arxiv cs.lg low mapped prevention quality regularization representation self-supervised learning space ssl state supervised learning type