Source: The Attacker Moves Second

Nasr, Carlini, Tramèr et al. (Oct 2025): adaptive attacks exceed 90% success against most of 12 published jailbreak and prompt-injection defences that reported near-zero ASR.

Contents
  • What: research paper, arXiv:2510.09023
  • Authors: Milad Nasr, Nicholas Carlini, Chawin Sitawarin, Sander V. Schulhoff, Jamie Hayes, Michael Ilie, Juliette Pluto, Shuang Song, Harsh Chaudhari, Ilia Shumailov, Abhradeep Thakurta, Kai Yuanqing Xiao, Andreas Terzis, Florian Tramèr
  • Date: 2025-10-10
  • Original: arXiv:2510.09023

Summary

Argues that LLM defences are evaluated wrongly: against fixed attack strings or weak optimisers not tailored to the defence. Re-evaluates 12 recent defences with scaled, adaptive attacks (gradient descent, reinforcement learning, random search, human-guided exploration).

Key claims

  • Attack success above 90% for most of the 12 defences, most of which originally reported near-zero ASR.
  • A human red-teaming competition (about 500 participants, $20,000 prize pool, per Willison’s summary) defeated every defence.
  • Robustness claims need adaptive attackers and substantial attack compute; near-zero ASR on static sets should be treated sceptically.

Feeds

Prompt injection · Jailbreaks

Suggest changes →