Zero-Shot Video Captioning with Evolving Pseudo-Tokens. (arXiv:2207.11100v2 [cs.CV] UPDATED)

We introduce a zero-shot video captioning method that employs two frozen
networks: the GPT-2 language model and the CLIP image-text matching model. The
matching score is used to steer the language model toward generating a sentence
that has a high average matching score to a subset of the video frames. Unlike
zero-shot image captioning methods, our work considers the entire sentence at
once. This is achieved by optimizing, during the generation process, part of
the prompt from scratch, by modifying …

arxiv captioning cv tokens video