The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

arXiv:2404.13208v1 Announce Type: cross
Abstract: Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly …

abstract arxiv attacks cs.cl cs.cr cs.lg llms prompt prompt injections prompts training training llms type vulnerabilities work