Back to blog

Hyperfence Blog

Memento: why your AI carries its own notes, and why anyone can rewrite them

By Simon Holliday

Memento: why your AI carries its own notes, and why anyone can rewrite them

Here is something that surprises most people the first time they see it, including engineers who have been building on these APIs for years.

When you send a message to an AI assistant, the model does not remember your previous messages. It cannot. What actually happens is that the app you are using sends the entire conversation again, from the beginning, every single time. Turn twenty means transmitting turns one through nineteen once more, along with the new thing you just typed.

The continuity is a performance, reassembled from scratch on every request out of a transcript that your side of the connection holds and your side of the connection sends back.

It is the arrangement Leonard Shelby lives by in Memento. Unable to form new memories, he carries his history around with him written down, on Polaroids and tattoos, and trusts it because it is the only history he has.

That arrangement has consequences, and the most important one is this: if the conversation is something your side sends, then the conversation is something your side can edit. What the model receives as “what you and I said earlier” is not a memory. It is a claim, arriving with the request, from whoever made the request.

Why the whole conversation goes back every time

The underlying reason is what a language model actually is. Strip away the chat window and it is a calculation: text goes in at one end, and a prediction of what should come next comes out at the other. Nothing about it changes in between, and nothing is set aside for later. There is no process running at the far end that holds your conversation in mind while it waits for your next message, in the way a phone call stays open between sentences. Each request is answered cold. So everything the model appears to know about your conversation has to arrive inside the request that needs it known.

The familiar structure of a conversation, a list of messages each labelled as coming from you or from the assistant, sits on top of that. It is a useful way to think about it and it is also a simplification, because before the model sees any of it that list is flattened into a single unbroken stretch of text. That flattening is where most of what follows comes from.

Armin Ronacher, writing in November 2025, argues that thinking of this as a simple exchange of messages misdescribes it, and that it is better understood as “a distributed state synchronization problem”, the kind where two systems each hold part of the truth and have to be kept in step. The provider’s part includes things your side never sees: hidden instructions, formatting markers, saved computation. He also states the cost plainly: “the size of each individual request grows linearly with the number of turns, but the cumulative amount of data sent over a long conversation grows quadratically because each linear-sized history is retransmitted at every step.” In ordinary terms, every message you add makes the next request bigger, and the total you have sent by the end of a long conversation is far larger than the conversation itself.

Which raises the obvious question. If this is so wasteful, why has nobody fixed it?

Largely because the expensive part was never sending the text. It is the model re-reading it, working out afresh how each part of the conversation bears on every other. That is the part which can be saved and reused. Serving systems keep the results of processing an opening stretch of conversation they have handled before and reuse them rather than starting from nothing, so what is stored is the work, not the words. That made the resend cheap enough to live with.

One detail of that shortcut is worth holding on to: the saved work can only be reused when the opening token sequence is identical. The economics silently assume that the history coming back is the history that went out. That assumption is about cost. Nothing enforces it.

Roles are a convention, not a boundary

If the client can edit the transcript, the obvious defence is to make sure the model can tell whose words are whose. That is exactly what the industry tried, and the attempt is older than most of the attacks.

When OpenAI introduced the chat format in 2023, the accompanying ChatML document explained the reasoning. The format “makes explicit to the model the source of each piece of text, and particularly shows the boundary between human and AI text”, and this “gives an opportunity to mitigate and eventually solve injections, as the model can tell which instructions come from the developer, the user, or its own input.” Each speaker was given reserved markers of their own, odd-looking symbols such as <|im_start|> and <|im_end|> that fence off where one turn begins and ends, so the boundary would be part of the structure rather than something the model had to guess at.

The remarkable part is what the same document says a little further on, about what happens when all of this is written out as ordinary text: “this format inherently allows injections from user input containing special-token syntax, similar to SQL injections.” It is the oldest trick in web security: if someone types those reserved markers into their message, the text they typed can be mistaken for the structure that was supposed to contain it.

The structure was introduced as a defence, and its designers wrote down the hole in the same breath. This is not a story about nobody thinking it through.

The next attempt was to make roles carry authority rather than merely identity. In April 2024, researchers at OpenAI published the instruction hierarchy, which begins from the observation that “LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties”, and proposes training models so that instructions from more privileged sources win when they conflict.

It helped. It did not hold. In February 2025, a separate group evaluating six leading models reported that “the widely-adopted system/user prompt separation fails to establish a reliable instruction hierarchy”. Their more uncomfortable finding was that appeals borrowed from ordinary social life, to authority or expertise or consensus, moved the models’ behaviour more reliably than the formal ranking of speakers did.

The mechanism underneath was named more recently still. Work presented at ICML 2026 frames the whole problem as role confusion, and its summary is hard to improve on: models “identify roles from an insecure feature (style)”, which the authors compare to “identifying a stranger’s profession from how they talk and dress rather than by checking their ID”. When the style and the labels disagree, style wins. Simply putting the words “User: ” in front of text buried inside a tool’s output measurably shifts the model’s sense of who is speaking.

Put those together and the picture is coherent. The role is not a boundary the model enforces. It is a hint the model interprets, and a forged turn does not need to defeat the format. It only needs to read like the real thing.

What this makes possible

The attacks follow from the design, which is why several of them were found independently within days of each other. They carry different names, but the research literature tends to file them under one idea: context manipulation, meaning changing the conversation on its way to the model rather than attacking the model itself.

The clearest is the Context Compliance Attack, published by Mark Russinovich at Microsoft in March 2025. The method needs no prompt engineering and no optimisation. You open a conversation on a sensitive subject. You then edit into the history a fabricated assistant turn, one that offers to help and asks whether the user would like the restricted material. Then you send a user turn that says yes. The model, reading a transcript in which it apparently already agreed, continues in character. It is Natalie’s trick, from the same film: knowing Leonard cannot retain what just happened, she can tell him whatever she likes about it, and his own notes will carry it forward as fact. Microsoft reported the attack succeeding on 10 of 11 sensitive task categories against Llama 3.1 at both sizes and Qwen 2.5-7b, and 9 of 11 against GPT-4o. Their framing is the important part: this “exploits a fundamental design decision in modern AI deployment” rather than any particular model’s training.

That same week, an unrelated academic group published the Dialogue Injection Attack, which needs nothing but the public chat API or a knowledge of the model’s chat template - the ordinary interface anyone can call. It reports state-of-the-art success against recent models including Llama-3.1 and GPT-4o, and bypasses five separate defence mechanisms. Two independent teams landing the same idea in the same fortnight tells you it is a property of the format rather than a bug in a product.

The scaled-up version arrived earlier, from a lab publishing an attack on its own models. Anthropic’s many-shot jailbreaking, from April 2024, fills the context with a long faux dialogue, up to hundreds of exchanges in which the assistant answers harmful questions willingly, before asking the real one. The more fabricated exchanges are added, the better it works, following the same steady curve as the ordinary and entirely benign trick of showing a model a few worked examples so that it copies the pattern. The detail worth dwelling on is what happened when they tried to train it away: fine-tuning “merely delayed the jailbreak”, raising the number of examples needed without breaking the relationship.

And the attacker does not have to be the user. Indirect prompt injection, described by Greshake and colleagues in early 2023, works by planting instructions in material the application will later fetch for itself: a document, a web page, the result of a search, or whatever comes back from one of the connectors that let an assistant reach outside systems on your behalf. The user is not malicious and their input is not tampered with. The hostile text arrives through the same door as everything else, because, as that paper puts it, LLM-integrated applications “blur the line between data and instructions”. Combine that with the user-tagging result above and the shape is complete: content arrives from a source nobody vetted, wearing the costume of a role nobody assigned it.

What can actually be done

It is tempting to conclude that the answer is to stop letting the client hold the transcript, and keep the conversation on the provider’s own servers instead. Microsoft’s write-up notes that products working that way, ChatGPT and Copilot among them, are not susceptible to this particular attack.

It is not, however, a general answer to prompt injection. The conversation can be intact and still contain untrusted material returned by searches, tools and connectors. That material still needs to be treated as data arriving from outside, not as trusted context.

One part of the transcript has recently stopped being freely editable, and it is worth knowing about. All three major providers now wrap the model’s private reasoning, the working-out it does before answering, in something your side carries but cannot read. Anthropic returns an encrypted copy of the full reasoning in a signature attached to each block of thinking, and is explicit about why: the API “uses the signature to verify that thinking blocks were generated by Claude when you pass them back”. OpenAI returns encrypted reasoning so that it can be carried between turns even when the provider keeps no copy of the conversation. Google’s thought signatures are “encrypted representations of the model’s internal thought process”, have to be handed back exactly as received, and on its latest models a missing one gets the request rejected outright rather than merely flagged.

Three different motivations, integrity, privacy and continuity, arriving at the same structure: a portion of the conversation that travels with the request, sealed against alteration and unreadable in transit. It is a genuine improvement, and it is also narrow. It covers the model’s reasoning, not the ordinary back-and-forth of user and assistant turns, which is where every attack described above actually operates. And that seal cuts both ways, because text nobody can read is text nobody can check, including whoever is meant to be inspecting it.

The more robust assumption is the pessimistic one. Treat the transcript as what it is, untrusted material arriving from outside, and check it accordingly.

In practice that means a few things. Check the whole history on every request rather than only the newest message, because something refused when it was new must not be able to return as background. Do that checking somewhere the person being governed cannot reach, which rules out the conversation itself. And inspect whatever arrives from searches, tools and outside connectors on the way in, not only the things a human typed, since that is where planted instructions enter for an assistant that reads the wider world.

That last set is where our own work sits, so briefly, for anyone wondering whether this is handled: Hyperfence checks every relevant turn of the history on every request rather than only the newest, neutralising anything that breaks policy on a copy before it is sent onward, and leaving clean turns untouched. It applies to what comes back from tools and connectors as well as to user messages. The practical effect is that content which would be refused as a fresh message cannot get in by arriving dressed as something said earlier. None of it depends on keeping your text: what is stored is a one-way fingerprint from which the original cannot be reconstructed, plus a note of the verdict, never the messages themselves.

If the machinery of that is interesting, we wrote up how a topic description and some example material become a detector with a measured error rate: How we build a Hyperfence.

None of which changes the underlying fact, and it is not going away. The model remembers nothing. Everything it appears to know about your conversation is in the request you just sent, which makes that request the thing worth reading carefully.

Sources