Deep Augmentation: Self-Supervised Learning with Transformations in Activation Space

arXiv:2303.14537v2 Announce Type: replace
Abstract: We introduce Deep Augmentation, an approach to implicit data augmentation using dropout or PCA to transform a targeted layer within a neural network to improve performance and generalization. We demonstrate Deep Augmentation through extensive experiments on contrastive learning tasks in NLP, computer vision, and graph learning. We observe substantial performance gains with Transformers, ResNets, and Graph Neural Networks as the underlying models in contrastive learning, but observe inverse effects on the corresponding supervised problems. Our …

abstract arxiv augmentation computer computer vision cs.cl cs.cv cs.lg data dropout graph graph learning layer network neural network nlp performance self-supervised learning space supervised learning tasks through type vision