VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. (arXiv:2105.04906v3 [cs.CV] UPDATED)

Web: http://arxiv.org/abs/2105.04906

Recent self-supervised methods for image representation learning are based on
maximizing the agreement between embedding vectors from different views of the
same image. A trivial solution is obtained when the encoder outputs constant
vectors. This collapse problem is often avoided through implicit biases in the
learning architecture, that often lack a clear justification or interpretation.
In this paper, we introduce VICReg (Variance-Invariance-Covariance
Regularization), a method that explicitly avoids the collapse problem with a
simple regularization term on the variance of …

arxiv cv learning self-supervised learning supervised learning variance