A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)

Contrastive Language-Image Pre-training (CLIP) models have demonstrated remarkable generalization capabilities across multiple challenging distribution shifts. However, there is still much to be explored in terms of their robustness to the variations of specific visual factors. In real-world applications, reliable and safe systems must consider other safety objectives beyond classification accuracy, such as predictive uncertainty. Yet, the effectiveness of CLIP models on such safety-related features is less-explored. Driven by the above, this work comprehensively investigates the safety objectives of CLIP models, …

applications beyond capabilities classification clip closer look cs.cv cs.lg distribution image language look multiple pre-training robustness safety systems terms training visual world