The Common Stability Mechanism behind most Self-Supervised Learning Approaches

arXiv:2402.14957v1 Announce Type: cross
Abstract: Last couple of years have witnessed a tremendous progress in self-supervised learning (SSL), the success of which can be attributed to the introduction of useful inductive biases in the learning process to learn meaningful visual representations while avoiding collapse. These inductive biases and constraints manifest themselves in the form of different optimization formulations in the SSL techniques, e.g. by utilizing negative examples in a contrastive formulation, or exponential moving average and predictor in BYOL and …

abstract arxiv biases constraints cs.cv cs.lg inductive introduction learn manifest process progress self-supervised learning ssl stability success supervised learning type visual