EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens

Masked Video Autoencoder (MVA) approaches have demonstrated their potential by significantly outperforming previous video representation learning methods. However, they waste an excessive amount of computations and memory in predicting uninformative tokens/frames due to random masking strategies. (e.g., over 16 nodes with 128 NVIDIA A100 GPUs). To resolve this issue, we exploit the unequal information density among the patches in videos and propose EVEREST, a surprisingly efficient MVA approach for video representation learning that finds tokens containing rich motion features and …

a100 autoencoder cs.cv cs.lg everest gpus issue masking memory nvidia nvidia a100 random representation representation learning strategies tokens video waste