vid-TLDR: Training Free Token merging for Light-weight Video Transformer

arXiv:2403.13347v1 Announce Type: new
Abstract: Video Transformers have become the prevalent solution for various video downstream tasks with superior expressive power and flexibility. However, these video transformers suffer from heavy computational costs induced by the massive number of tokens across the entire video frames, which has been the major barrier to training the model. Further, the patches irrelevant to the main contents, e.g., backgrounds, degrade the generalization performance of models. To tackle these issues, we propose training free token merging …

arxiv cs.cv free light merging token training transformer type video