Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations

arXiv:2403.07241v1 Announce Type: cross
Abstract: Fine-tuning pre-trained vision-language models, like CLIP, has yielded success on diverse downstream tasks. However, several pain points persist for this paradigm: (i) directly tuning entire pre-trained models becomes both time-intensive and computationally costly. Additionally, these tuned models tend to become highly specialized, limiting their practicality for real-world deployment; (ii) recent studies indicate that pre-trained vision-language classifiers may overly depend on spurious features -- patterns that correlate with the target in training data, but are not …

abstract annotations arxiv become clip cs.cv cs.lg diverse fine-tuning however language language models modal multi-modal pain paradigm pre-trained models robustness success tasks type vision vision-language models