Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks

arXiv:2405.04403v1 Announce Type: new
Abstract: Augmenting Large Language Models (LLMs) with image-understanding capabilities has resulted in a boom of high-performing Vision-Language models (VLMs). While studying the alignment of LLMs to human values has received widespread attention, the safety of VLMs has not received the same attention. In this paper, we explore the impact of jailbreaking on three state-of-the-art VLMs, each using a distinct modeling approach. By comparing each VLM to their respective LLM backbone, we find that each VLM is …

abstract alignment arxiv attacks attention boom capabilities cs.cv human image instruction tuning jailbreak language language models large language large language models llms safety studying type understanding values vision vision-language vision-language models visual vlms while