Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study

arXiv:2403.10499v1 Announce Type: new
Abstract: Pre-training image representations from the raw text about images enables zero-shot vision transfer to downstream tasks. Through pre-training on millions of samples collected from the internet, multimodal foundation models, such as CLIP, produce state-of-the-art zero-shot results that often reach competitiveness with fully supervised methods without the need for task-specific training. Besides the encouraging performance on classification accuracy, it is reported that these models close the robustness gap by matching the performance of supervised models trained …

abstract art arxiv benchmarking clip cs.ai cs.cl cs.cv cs.lg foundation image images internet multimodal pilot pre-training raw results robustness samples state study tasks text through training transfer type vision zero-shot