The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning. (arXiv:2205.06226v1 [cs.LG])

Recently the surprising discovery of the Bootstrap Your Own Latent (BYOL)
method by Grill et al. shows the negative term in contrastive loss can be
removed if we add the so-called prediction head to the network. This initiated
the research of non-contrastive self-supervised learning. It is mysterious why
even when there exist trivial collapsed global optimal solutions, neural
networks trained by (stochastic) gradient descent can still learn competitive
representations. This phenomenon is a typical example of implicit bias in deep …

arxiv head learning prediction self-supervised learning supervised learning