Text-Conditioned Resampler For Long Form Video Understanding

arXiv:2312.11897v2 Announce Type: replace
Abstract: In this paper we present a text-conditioned video resampler (TCR) module that uses a pre-trained and frozen visual encoder and large language model (LLM) to process long video sequences for a task. TCR localises relevant visual features from the video given a text condition and provides them to a LLM to generate a text response. Due to its lightweight design and use of cross-attention, TCR can process more than 100 frames at a time with …

abstract arxiv cs.cv encoder features form language language model large language large language model llm paper process text them type understanding video video understanding visual