Boosting Visual-Language Models by Exploiting Hard Samples

arXiv:2305.05208v2 Announce Type: replace
Abstract: Contrastive Language-Image Pre-training (CLIP) has become the standard for learning cross-modal representations between images and text. Efforts to improve its capabilities typically demand the collection of additional data and retraining with new loss functions. While effective, the added requirements limit their practical use due to the increased resource and time investments needed. In this work, we present HELIP, a cost-effective strategy tailored to enhance the performance of existing CLIP models without the need for training …

abstract arxiv become boosting capabilities clip collection cs.cv data demand functions image images language language models loss modal practical pre-training requirements retraining samples standard text training type visual