Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation

arXiv:2403.12042v1 Announce Type: new
Abstract: In this paper, we explore the visual representations produced from a pre-trained text-to-video (T2V) diffusion model for video understanding tasks. We hypothesize that the latent representation learned from a pretrained generative T2V model encapsulates rich semantics and coherent temporal correspondences, thereby naturally facilitating video understanding. Our hypothesis is validated through the classic referring video object segmentation (R-VOS) task. We introduce a novel framework, termed ``VD-IT'', tailored with dedicatedly designed components built upon a fixed pretrained …

arxiv cs.cv diffusion diffusion models object segmentation text text-to-video type video video diffusion