TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning. (arXiv:2206.10698v2 [cs.CV] UPDATED)

We present Transformation Invariance and Covariance Contrast (TiCo) for
self-supervised visual representation learning. Similar to other recent
self-supervised learning methods, our method is based on maximizing the
agreement among embeddings of different distorted versions of the same image,
which pushes the encoder to produce transformation invariant representations.
To avoid the trivial solution where the encoder generates constant vectors, we
regularize the covariance matrix of the embeddings from different images by
penalizing low rank solutions. By jointly minimizing the transformation
invariance …

arxiv covariance cv learning representation representation learning transformation