Improving Adversarial Transferability of Visual-Language Pre-training Models through Collaborative Multimodal Interaction

arXiv:2403.10883v1 Announce Type: new
Abstract: Despite the substantial advancements in Vision-Language Pre-training (VLP) models, their susceptibility to adversarial attacks poses a significant challenge. Existing work rarely studies the transferability of attacks on VLP models, resulting in a substantial performance gap from white-box attacks. We observe that prior work overlooks the interaction mechanisms between modalities, which plays a crucial role in understanding the intricacies of VLP models. In response, we propose a novel attack, called Collaborative Multimodal Interaction Attack (CMI-Attack), leveraging …

abstract adversarial adversarial attacks arxiv attacks box challenge collaborative cs.cr cs.cv cs.mm gap language multimodal observe performance pre-training prior studies through training training models type vision visual work