What to Hide from Your Students: Attention-Guided Masked Image Modeling. (arXiv:2203.12719v2 [cs.CV] UPDATED)

Transformers and masked language modeling are quickly being adopted and
explored in computer vision as vision transformers and masked image modeling
(MIM). In this work, we argue that image token masking differs from token
masking in text, due to the amount and correlation of tokens in an image. In
particular, to generate a challenging pretext task for MIM, we advocate a shift
from random masking to informed masking. We develop and exhibit this idea in
the context of distillation-based MIM, …

arxiv attention cv hide image modeling students