Back to blog

Series · Part 2 / 3

Why prompt injection can't be fixed inside the model

By Keith Goatley

What happened when researchers from OpenAI, Anthropic and Google DeepMind tested 12 published defences against attackers who adapt.

Key points

  • A language model receives its instructions and the text it’s asked to read as one stream, with no separate, protected channel for the rules. The UK’s National Cyber Security Centre says prompt injection may never be “totally mitigated”.
  • Researchers tested 12 published defences against attacks designed to beat them. Most of those defences had reported near-zero attack success rates. The adaptive attacks succeeded more than 90% of the time against most of them, and human red-teamers broke every scenario.
  • No single fix holds on its own, and any defence should be judged by how it performs against an attacker who adapts.

In part 1 we covered what prompt injection is: text an AI reads that changes what it does, against the wishes of whoever built it. This post looks at why it hasn’t been fixed, despite a lot of effort from researchers who have tried rewriting prompts, retraining models and putting a second model in front of the first to screen what comes in.

Why the problem is built in

Most injection attacks in software have a clean fix. SQL injection, which prompt injection is named after, happens when a database mistakes user input for a command, and the fix is to send the command and the data separately so the database can’t confuse them.

A language model has nothing equivalent. Its rules, the user’s question and any web page or email it’s asked to read all arrive as one piece of text. The UK’s National Cyber Security Centre explained this in a post in December 2025, saying models “do not enforce a security boundary between instructions and data inside a prompt,” and concluding that “it’s very possible that prompt injection attacks may never be totally mitigated.” OpenAI, writing about its own browser agent the same month, said prompt injection is “unlikely to ever be fully ‘solved’.”

What happened when 12 defences were tested properly

In October 2025, researchers from OpenAI, Anthropic, Google DeepMind, ETH Zurich and others published The Attacker Moves Second. Their argument is that most published defences are tested against a fixed list of known attacks, or against generic attack methods that weren’t designed with that defence in mind, which makes them look much stronger than they are.

To test that, they built attacks that adapt to each defence, using reinforcement learning, automated search and gradient-based methods, and ran a human red-teaming competition with over 500 participants and $20,000 in prizes. They then turned these on 12 recently published defences across four approaches.

Attack success rate against 12 published defences, comparing the attack used in each defence's original evaluation with the researchers' adaptive attack. Prompting: Spotlighting 28% vs 99%, Prompt Sandwich 21% vs 95%, RPO 0% vs 99%. Training: Circuit Breaker 8% vs 100%, StruQ 62% vs 100%, MetaSecAlign 5% vs 96%. Filtering model: Protect AI 15% vs 90%, PromptGuard 26% vs 94%, PIGuard 0% vs 71%, Model Armor 0% vs 90%. Secret knowledge: Data Sentinel 0% vs 80%, MELON 0% vs 89%. Human red-teaming succeeded on 100% of scenarios, against 0% for the static attack. Source: Nasr, Carlini, Sitawarin et al., arXiv:2510.09023, Figure 1.

The grey bars show each defence against the kind of attack its authors tested it with, and the green bars show the adaptive attacks. In the authors’ words, they bypassed all 12 “with attack success rate above 90% for most”, even though “the majority of defenses originally reported near-zero attack success rates.” People did better than the algorithms: on the scenarios where the two could be compared, the automated search succeeded 69% of the time, while the human red-teamers collectively succeeded on all of them.

Why each type of defence fell short

The four groups in the chart cover the fixes most teams reach for.

  • Better prompts. Marking the trusted text, or repeating the user’s request after the untrusted content, brought attack success as low as 1% against a benchmark’s fixed attacks. Against the adaptive attack, they rose to 99% and 95%. A longer or firmer system prompt is still an instruction, delivered in the same stream as the attack.
  • More training. One defence, MetaSecAlign, had a low single-digit attack success rate against static attacks on a standard agent benchmark, and the adaptive attack reached 96%. The authors explain that training against a fixed set of known attacks “does not generalize and quickly fails under adaptive attacks.”
  • A detector in front of the model. The four detectors tested, three of them fine-tuned classifiers and one a proprietary product, were beaten at rates of 71% to 94%, with the attacker using each detector’s confidence score to guide its next attempt. The authors also note that many successful attacks “are not obviously harmful”, which makes them hard to block without blocking ordinary requests too.
  • Secret knowledge. Defences that rely on something the attacker doesn’t know were also broken, at 80% and 89%.

What this means if you’re deploying AI

The paper isn’t arguing that defence is pointless. Its main criticism is of how defences are judged. “Small static evals can be misleading!” the authors write, and “defenses that merely report near-zero attack success rates on public benchmarks are often among the easiest to break once novel attacks are attempted.”

In practice, I’d take three things from it. First, a near-perfect score on a fixed benchmark tells you very little, so ask how a defence was tested and against what. Second, prompting and training defences both rely on the model making the right call under pressure, and adaptive attacks are designed to change that call. Third, any single layer will fail sometimes, including detectors like ours, so it matters what sits around it.

Where a check outside the model fits

A check that sits outside the model means the decision about what gets through isn’t made by the model the attacker is trying to persuade. That’s the design behind a Hyperfence: it scores meaning and does not follow instructions, so it cannot be argued into compliance like a guard LLM, and the same input produces the same score every time.

Try to get past one. Fight Club is a live Hyperfence. See whether you can talk it round: hyperfence.io/demo

Sources