Typographic Attacks in Large Multimodal Models Can be Alleviated by More Informative Prompts

arXiv:2402.19150v1 Announce Type: new
Abstract: Large Multimodal Models (LMMs) rely on pre-trained Vision Language Models (VLMs) and Large Language Models (LLMs) to perform amazing emergent abilities on various multimodal tasks in the joint space of vision and language. However, the Typographic Attack, which shows disruption to VLMs, has also been certified as a security vulnerability to LMMs. In this work, we first comprehensively investigate the distractibility of LMMs by typography. In particular, we introduce the Typographic Dataset designed to evaluate …

abstract arxiv attacks cs.cv disruption language language models large language large language models large multimodal models llms lmms multimodal multimodal models prompts shows space tasks type vision vlms