RankCLIP: Ranking-Consistent Language-Image Pretraining

arXiv:2404.09387v1 Announce Type: cross
Abstract: Among the ever-evolving development of vision-language models, contrastive language-image pretraining (CLIP) has set new benchmarks in many downstream tasks such as zero-shot classifications by leveraging self-supervised contrastive learning on large amounts of text-image pairs. However, its dependency on rigid one-to-one mappings overlooks the complex and often multifaceted relationships between and within texts and images. To this end, we introduce RankCLIP, a novel pretraining method that extends beyond the rigid one-to-one matching framework of CLIP and …

abstract arxiv benchmarks clip consistent cs.ai cs.cv cs.lg development ever however image language language models pretraining ranking relationships set tasks text text-image type vision vision-language models zero-shot