Adversarial Illusions in Multi-Modal Embeddings

arXiv:2308.11804v3 Announce Type: replace-cross
Abstract: Multi-modal embeddings encode texts, images, sounds, videos, etc., into a single embedding space, aligning representations across different modalities (e.g., associate an image of a dog with a barking sound). In this paper, we show that multi-modal embeddings can be vulnerable to an attack we call "adversarial illusions." Given an image or a sound, an adversary can perturb it to make its embedding close to an arbitrary, adversary-chosen input in another modality.
These attacks are cross-modal …

abstract adversarial arxiv call cs.ai cs.cr cs.lg dog embedding embeddings encode etc image images modal multi-modal paper show sound space type videos vulnerable