Learning text-to-video retrieval from image captioning

arXiv:2404.17498v1 Announce Type: new
Abstract: We describe a protocol to study text-to-video retrieval training with unlabeled videos, where we assume (i) no access to labels for any videos, i.e., no access to the set of ground-truth captions, but (ii) access to labeled images in the form of text. Using image expert models is a realistic scenario given that annotating images is cheaper therefore scalable, in contrast to expensive video labeling schemes. Recently, zero-shot image experts such as CLIP have established …

abstract access arxiv captioning captions cs.cv expert form ground-truth image images labels protocol retrieval set study text text-to-video training truth type video videos