ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding. (arXiv:2109.04380v2 [cs.CL] UPDATED)

Contrastive learning has been attracting much attention for learning
unsupervised sentence embeddings. The current state-of-the-art unsupervised
method is the unsupervised SimCSE (unsup-SimCSE). Unsup-SimCSE takes dropout as
a minimal data augmentation method, and passes the same input sentence to a
pre-trained Transformer encoder (with dropout turned on) twice to obtain the
two corresponding embeddings to build a positive pair. As the length
information of a sentence will generally be encoded into the sentence
embeddings due to the usage of position embedding …

arxiv building embedding unsupervised