Positional Label for Self-Supervised Vision Transformer. (arXiv:2206.04981v2 [cs.CV] UPDATED)

Positional encoding is important for vision transformer (ViT) to capture the
spatial structure of the input image. General effectiveness has been proven in
ViT. In our work we propose to train ViT to recognize the positional label of
patches of the input image, this apparently simple task actually yields a
meaningful self-supervisory task. Based on previous work on ViT positional
encoding, we propose two positional labels dedicated to 2D images including
absolute position and relative position. Our positional labels can …

arxiv cv transformer vision