Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. (arXiv:2106.04156v7 [cs.LG] UPDATED)

Recent works in self-supervised learning have advanced the state-of-the-art
by relying on the contrastive learning paradigm, which learns representations
by pushing positive pairs, or similar examples from the same class, closer
together while keeping negative pairs far apart. Despite the empirical
successes, theoretical foundations are limited -- prior analyses assume
conditional independence of the positive pairs given the same class label, but
recent empirical applications use heavily correlated positive pairs (i.e., data
augmentations of the same image). Our work analyzes …

arxiv deep learning learning lg loss