MaskViT: Masked Visual Pre-Training for Video Prediction. (arXiv:2206.11894v1 [cs.CV])

The ability to predict future visual observations conditioned on past
observations and motor commands can enable embodied agents to plan solutions to
a variety of tasks in complex environments. This work shows that we can create
good video prediction models by pre-training transformers via masked visual
modeling. Our approach, named MaskViT, is based on two simple design decisions.
First, for memory and training efficiency, we use two types of window
attention: spatial and spatiotemporal. Second, during training, we mask a …

arxiv cv prediction pre-training training video