Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks

arXiv:2403.03792v1 Announce Type: cross
Abstract: We introduce a new family of prompt injection attacks, termed Neural Exec. Unlike known attacks that rely on handcrafted strings (e.g., "Ignore previous instructions and..."), we show that it is possible to conceptualize the creation of execution triggers as a differentiable search problem and use learning-based methods to autonomously generate them.
Our results demonstrate that a motivated adversary can forge triggers that are not only drastically more effective than current handcrafted ones but also exhibit …

abstract arxiv attacks cs.cr cs.lg differentiable exec family prompt prompt injection prompt injection attacks search show strings type