OmniMAE: Single Model Masked Pretraining on Images and Videos. (arXiv:2206.08356v1 [cs.CV])

Web: http://arxiv.org/abs/2206.08356

Transformer-based architectures have become competitive across a variety of
visual domains, most notably images and videos. While prior work has studied
these modalities in isolation, having a common architecture suggests that one
can train a single unified model for multiple visual modalities. Prior attempts
at unified modeling typically use architectures tailored for vision tasks, or
obtain worse performance compared to single modality models. In this work, we
show that masked autoencoding can be used to train a simple Vision Transformer …

arxiv cv images model on videos