InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models

arXiv:2312.01886v3 Announce Type: replace
Abstract: Large vision-language models (LVLMs) have demonstrated their incredible capability in image understanding and response generation. However, this rich visual interaction also makes LVLMs vulnerable to adversarial examples. In this paper, we formulate a novel and practical targeted attack scenario that the adversary can only know the vision encoder of the victim LVLM, without the knowledge of its prompts (which are often proprietary for service providers and not publicly available) and its underlying large language model …

arxiv cs.cv instruction-tuned language language models replace type vision vision-language vision-language models