SdAE: Self-distillated Masked Autoencoder. (arXiv:2208.00449v1 [cs.CV])

With the development of generative-based self-supervised learning (SSL)
approaches like BeiT and MAE, how to learn good representations by masking
random patches of the input image and reconstructing the missing information
has grown in concern. However, BeiT and PeCo need a "pre-pretraining" stage to
produce discrete codebooks for masked patches representing. MAE does not
require a pre-training codebook process, but setting pixels as reconstruction
targets may introduce an optimization gap between pre-training and downstream
tasks that good reconstruction quality may …

arxiv autoencoder cv masked autoencoder