Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective

arXiv:2404.19287v1 Announce Type: new
Abstract: Pretrained vision-language models (VLMs) like CLIP have shown impressive generalization performance across various downstream tasks, yet they remain vulnerable to adversarial attacks. While prior research has primarily concentrated on improving the adversarial robustness of image encoders to guard against attacks on images, the exploration of text-based and multimodal attacks has largely been overlooked. In this work, we initiate the first known and comprehensive effort to study adapting vision-language models for adversarial robustness under the multimodal …

abstract adversarial adversarial attacks arxiv attacks clip cs.cv exploration image images improving language language models multimodal performance perspective prior research robustness tasks type vision vision-language vision-language models vlms vulnerable while