Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation

arXiv:2403.09572v1 Announce Type: new
Abstract: Multimodal large language models (MLLMs) have shown impressive reasoning abilities, which, however, are also more vulnerable to jailbreak attacks than their LLM predecessors. Although still capable of detecting unsafe responses, we observe that safety mechanisms of the pre-aligned LLMs in MLLMs can be easily bypassed due to the introduction of image features. To construct robust MLLMs, we propose ECSO(Eyes Closed, Safety On), a novel training-free protecting approach that exploits the inherent safety awareness of MLLMs, …

abstract arxiv attacks cs.cv however image image-to-text jailbreak language language models large language large language models llm llms mllms multimodal multimodal llms observe reasoning responses safety text transformation type via vulnerable