Analysis of Using Sigmoid Loss for Contrastive Learning

arXiv:2402.12613v1 Announce Type: new
Abstract: Contrastive learning has emerged as a prominent branch of self-supervised learning for several years. Especially, CLIP, which applies contrastive learning to large sets of captioned images, has garnered significant attention. Recently, SigLIP, a variant of CLIP, has been proposed, which uses the sigmoid loss instead of the standard InfoNCE loss. SigLIP achieves the performance comparable to CLIP in a more efficient manner by eliminating the need for a global view. However, theoretical understanding of using …

abstract analysis arxiv attention clip cs.lg images loss self-supervised learning sigmoid supervised learning type