Contrastive Learning with Orthonormal Anchors (CLOA)

arXiv:2403.18699v1 Announce Type: new
Abstract: This study focuses on addressing the instability issues prevalent in contrastive learning, specifically examining the InfoNCE loss function and its derivatives. We reveal a critical observation that these loss functions exhibit a restrictive behavior, leading to a convergence phenomenon where embeddings tend to merge into a singular point. This "over-fusion" effect detrimentally affects classification accuracy in subsequent supervised-learning tasks. Through theoretical analysis, we demonstrate that embeddings, when equalized or confined to a rank-1 linear subspace, …

abstract anchors arxiv behavior convergence cs.ai cs.lg derivatives embeddings function functions loss merge observation restrictive singular study type