Source: The Instruction Hierarchy (OpenAI)

Wallace et al. (2024): train models to prioritise privileged (system/developer) instructions over lower-privileged text to resist injection.

Contents
  • What: research paper, arXiv:2404.13208
  • Authors: Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, Alex Beutel (OpenAI)
  • Date: 2024-04-19
  • Original: arXiv:2404.13208

Summary

Argues that prompt injection and many jailbreaks work because models treat a developer’s system prompt and untrusted user or third-party text as equally authoritative. The paper defines an instruction hierarchy and a synthetic-data method that trains the model to ignore lower-privileged instructions that conflict with higher-privileged ones.

Key claims

  • Privilege order: system message (from the application developer) highest, then user messages, then model outputs and tool outputs lowest.
  • Equal treatment of all instruction sources is a root cause of injections and jailbreaks.
  • A defined hierarchy says how to behave when instructions of different privilege conflict.
  • Training GPT-3.5 on generated data to selectively ignore lower-privileged instructions substantially improved robustness, including on attack types not seen in training, with minimal capability loss.

Feeds

System prompts · Prompt injection · Jailbreaks

Suggest changes →