Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition

arXiv:2311.15619v2 Announce Type: replace
Abstract: Large-scale visual-language pre-trained models have achieved significant success in various video tasks. However, most existing methods follow an "adapt then align" paradigm, which adapts pre-trained image encoders to model video-level representations and utilizes one-hot or text embedding of the action labels for supervision. This paradigm overlooks the challenge of mapping from static images to complicated activity concepts. In this paper, we propose a novel "Align before Adapt" (ALT) paradigm. Prior to adapting to video representation …

abstract action recognition adapt arxiv cs.ai cs.cv embedding hot however image labels language paradigm pre-trained models recognition scale success supervision tasks text text embedding type video visual