A simple, efficient and scalable contrastive masked autoencoder for learning visual representations. (arXiv:2210.16870v1 [cs.CV])

We introduce CAN, a simple, efficient and scalable method for self-supervised
learning of visual representations. Our framework is a minimal and conceptually
clean synthesis of (C) contrastive learning, (A) masked autoencoders, and (N)
the noise prediction approach used in diffusion models. The learning mechanisms
are complementary to one another: contrastive learning shapes the embedding
space across a batch of image samples; masked autoencoders focus on
reconstruction of the low-frequency spatial correlations in a single image
sample; and noise prediction encourages …

arxiv autoencoder masked autoencoder scalable