RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training. (arXiv:2201.06857v2 [cs.CV] UPDATED)

Recently, self-supervised vision transformers have attracted unprecedented
attention for their impressive representation learning ability. However, the
dominant method, contrastive learning, mainly relies on an instance
discrimination pretext task, which learns a global understanding of the image.
This paper incorporates local feature learning into self-supervised vision
transformers via Reconstructive Pre-training (RePre). Our RePre extends
contrastive frameworks by adding a branch for reconstructing raw image pixels
in parallel with the existing contrastive objective. RePre is equipped with a
lightweight convolution-based decoder that …

arxiv cv training transformer vision