Transformers Provably Learn Feature-Position Correlations in Masked Image Modeling

arXiv:2403.02233v1 Announce Type: new
Abstract: Masked image modeling (MIM), which predicts randomly masked patches from unmasked ones, has emerged as a promising approach in self-supervised vision pretraining. However, the theoretical understanding of MIM is rather limited, especially with the foundational architecture of transformers. In this paper, to the best of our knowledge, we provide the first end-to-end theory of learning one-layer transformers with softmax attention in MIM self-supervised pretraining. On the conceptual side, we posit a theoretical mechanism of how …

abstract architecture arxiv best of correlations cs.lg feature image learn math.oc modeling paper pretraining stat.ml transformers type understanding vision