Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradient

arXiv:2402.18281v1 Announce Type: new
Abstract: Sentence Representation Learning (SRL) is a crucial task in Natural Language Processing (NLP), where contrastive Self-Supervised Learning (SSL) is currently a mainstream approach. However, the reasons behind its remarkable effectiveness remain unclear. Specifically, in other research fields, contrastive SSL shares similarities in both theory and practical performance with non-contrastive SSL (e.g., alignment & uniformity, Barlow Twins, and VICReg). However, in SRL, contrastive SSL outperforms non-contrastive SSL significantly. Therefore, two questions arise: First, what commonalities enable …

abstract arxiv cs.cl fields gradient language language processing natural natural language natural language processing nlp paradigm processing representation representation learning research self-supervised learning shares ssl supervised learning type understanding