Enhancing Video Transformers for Action Understanding with VLM-aided Training

arXiv:2403.16128v1 Announce Type: new
Abstract: Owing to their ability to extract relevant spatio-temporal video embeddings, Vision Transformers (ViTs) are currently the best performing models in video action understanding. However, their generalization over domains or datasets is somewhat limited. In contrast, Visual Language Models (VLMs) have demonstrated exceptional generalization performance, but are currently unable to process videos. Consequently, they cannot extract spatio-temporal patterns that are crucial for action understanding. In this paper, we propose the Four-tiered Prompts (FTP) framework that takes …

abstract arxiv contrast cs.cv datasets domains embeddings extract however language language models performance temporal training transformers type understanding video vision vision transformers visual visual language models vlm vlms