SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learning

arXiv:2210.03963v2 Announce Type: replace
Abstract: Contrastive learning has recently achieved compelling performance in unsupervised sentence representation. As an essential element, data augmentation protocols, however, have not been well explored. The pioneering work SimCSE resorting to a simple dropout mechanism (viewed as continuous augmentation) surprisingly dominates discrete augmentations such as cropping, word deletion, and synonym replacement as reported. To understand the underlying rationales, we revisit existing approaches and attempt to hypothesize the desiderata of reasonable data augmentation methods: balance of semantic …

abstract arxiv augmentation continuous cs.cl data dropout element however performance representation representation learning simple type unsupervised word work