Fine-tuning CLIP Text Encoders with Two-step Paraphrasing

arXiv:2402.15120v1 Announce Type: cross
Abstract: Contrastive language-image pre-training (CLIP) models have demonstrated considerable success across various vision-language tasks, such as text-to-image retrieval, where the model is required to effectively process natural language input to produce an accurate visual output. However, current models still face limitations in dealing with linguistic variations in input queries, such as paraphrases, making it challenging to handle a broad range of user queries in real-world applications. In this study, we introduce a straightforward fine-tuning approach to …

abstract arxiv clip cs.ai cs.cv cs.lg current face fine-tuning image language limitations natural natural language paraphrasing pre-training process queries retrieval success tasks text text-to-image training type vision visual