Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition

arXiv:2311.16119v3 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) are deployed in interactive contexts with direct user engagement, such as chatbots and writing assistants. These deployments are vulnerable to prompt injection and jailbreaking (collectively, prompt hacking), in which models are manipulated to ignore their original instructions and follow potentially malicious ones. Although widely acknowledged as a significant security threat, there is a dearth of large-scale resources and quantitative studies on prompt hacking. To address this lacuna, we launch a global …

abstract arxiv assistants chatbots competition cs.ai cs.cl cs.cr deployments engagement global hacking interactive jailbreaking language language models large language large language models llms prompt prompt injection scale through type user engagement vulnerabilities vulnerable writing