VideoPrism: A Foundational Visual Encoder for Video Understanding

arXiv:2402.13217v1 Announce Type: new
Abstract: We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated …

abstract arxiv asr cs.ai cs.cv diverse encoder general pretraining quality tasks text transcripts type understanding video video understanding visual