When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?

arXiv:2403.08673v1 Announce Type: new
Abstract: Contrastive learning is a paradigm for learning representations from unlabelled data that has been highly successful for image and text data. Several recent works have examined contrastive losses to claim that contrastive models effectively learn spectral embeddings, while few works show relations between (wide) contrastive models and kernel principal component analysis (PCA). However, it is not known if trained contrastive models indeed correspond to kernel methods or PCA. In this work, we analyze the training …

abstract analysis arxiv claim cs.lg data embeddings image learn losses paradigm stat.ml text type