Motion Guided Token Compression for Efficient Masked Video Modeling

arXiv:2402.18577v1 Announce Type: new
Abstract: Recent developments in Transformers have achieved notable strides in enhancing video comprehension. Nonetheless, the O($N^2$) computation complexity associated with attention mechanisms presents substantial computational hurdles when dealing with the high dimensionality of videos. This challenge becomes particularly pronounced when striving to increase the frames per second (FPS) to enhance the motion capturing capabilities. Such a pursuit is likely to introduce redundancy and exacerbate the existing computational limitations. In this paper, we initiate by showcasing the …

abstract arxiv attention attention mechanisms challenge complexity compression computation computational cs.ai cs.cv dimensionality modeling per token transformers type video videos