VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models

arXiv:2402.13851v1 Announce Type: new
Abstract: Autoregressive Visual Language Models (VLMs) showcase impressive few-shot learning capabilities in a multimodal context. Recently, multimodal instruction tuning has been proposed to further enhance instruction-following abilities. However, we uncover the potential threat posed by backdoor attacks on autoregressive VLMs during instruction tuning. Adversaries can implant a backdoor by injecting poisoned samples with triggers embedded in instructions or images, enabling malicious manipulation of the victim model's predictions with predefined triggers. Nevertheless, the frozen visual encoder in …

abstract arxiv attacks backdoor capabilities context cs.cv few-shot few-shot learning implant language language models multimodal threat type visual vlms