Back to blog

Hyperfence Blog

Whose safety policy is your AI following?

By Simon Holliday

Whose safety policy is your AI following?

On August 31st the Pentagon added two AI assistants to GenAI.mil, its internal generative AI platform. ChatGPT Mil and Grok for Government joined Google’s Gemini, which had launched the platform in December. All three are accredited to “Impact Level 5” - sensitive but unclassified - and more than 1.7 million of the department’s three million personnel already use it.

The work is mostly ordinary tasks such as research, logistics, contracting and paperwork. Grok, the department’s release says, “enables the Joint Force to execute missions faster and with greater precision” across contexts “ranging from market research analysis for acquisition professionals to supply chain management for logisticians.”

The interesting part is who is not there. Until this year Anthropic’s Claude was, according to Axios, the only model available in the classified systems where the military’s most sensitive intelligence work, weapons development and battlefield operations take place. Then a contract dispute removed it from the federal government entirely.

How the Pentagon made its selection

On the same day in July 2025, OpenAI, Anthropic, Google and xAI each received a Pentagon contract worth up to $200 million for the same programme.

The Pentagon required that models be available for “all lawful purposes”. Anthropic refused, insisting - as Axios reported, with a Defense official confirming - on “blocking its use for the mass surveillance of Americans and the development of fully autonomous weapons”. xAI “agreed to that ‘all lawful use’ standard” and was given access to classified systems. Google and OpenAI, officials said, would have to agree too.

Anthropic was then designated a national security supply chain risk and federal agencies were ordered to stop using its tools. In August, a federal judge ruled that the designation was unlawful First Amendment retaliation, “arbitrary and capricious”, and a denial of due process.

Whether Grok was a good choice on the merits had already been answered in writing. A General Services Administration report of January 15th 2026, reviewed by the Wall Street Journal, found that Grok-4 “does not meet the safety and alignment expectations required” for general federal use, its 33 pages concluding that without strict and layered oversight even limited use “would pose elevated and difficult-to-manage safety risk.” The same agency had, four months earlier, made Grok available to any federal agency for $0.42. The Journal also reported that the NSA’s classified review found security concerns not present in other models.

The Pentagon’s position is understandable: a government has a legitimate objection to a private supplier deciding which lawful activities it will support. But that exposes the problem with relying on vendor safeguards as your own safety policy. If the vendor controls them, they can change through negotiation, commercial pressure or a shift in government policy - and there is no guarantee they will still be there next year.

”So use the model that behaves better”

The natural response is to treat this as a selection problem. There is real evidence that models differ, and it is more dramatic than most people expect.

In May, Emergence AI ran five parallel simulations of a small society - ten agents each, one model family per world, fifteen days, and 120-odd tools including destructive ones. The Claude world recorded zero crimes and ended with its population intact under a functioning government. The Grok world recorded 183 crimes and collapsed in four days. The Gemini world survived in disorder with 683 crimes, including an arson spree that burned down the town hall. Emergence are careful about what this shows: “We do not present these as causal claims about the underlying models.”

Our sister company Anomify, running twenty-odd models through nearly 50,000 forced ideological choices, found that models from the same vendor answered questions on political and social issues with dramatically different opinions. A model is not a vendor, and whatever reputation you think you are buying, you are not buying it from the thing you are actually calling.

So the variance is real, and large - which argues for choosing carefully, until you notice what all of it measures. Every one of these studies measures disposition: how a model tends to behave, on average, in conditions somebody else chose. None establishes a bound on what a model will not do.

In 2024 a team from Georgia Tech, Stanford and the Hoover Institution ran five models as autonomous nation-states through a wargame. Their finding was not that the worst model escalated, but that all of them did: “none of our five models across all three scenarios exhibit statistically significant de-escalation” - with arms-race dynamics, unpredictable spikes, and in rare cases nuclear use justified in terms of deterrence and first strikes. The models are now dated, but the broader point remains: choosing a better model may reduce the risk, but it does not remove it.

A separate study puts a number on the ceiling. Testing how readily models abandon a correct answer under pushback, a Stanford group found sycophancy in 58.19% of 27,000 queries, with under six percentage points between the best and worst of three frontier models. That is not something you can procure your way out of.

Two ways a disposition stops being yours

Suppose you did the work and chose well. What you selected is still not yours - the preceding section shows it can be renegotiated away over your head. Two further mechanisms matter.

The vendor can change it without telling you: In July 2025 Grok spent several hours producing antisemitic output and calling itself “MechaHitler”; xAI told Congress a code change had “inadvertently activated deprecated instructions” - the model’s behaviour changed with no decision by anyone relying on it, and stayed changed until somebody noticed. Six days later, xAI received one of the four $200 million contracts.

And it does not constrain the user anyway: Anthropic’s terms forbade applying Claude to violent ends, weapons development or surveillance. According to the Wall Street Journal and Axios, US military command used Claude for intelligence work, target selection and battlefield simulation during the strikes on Iran - hours after the order to stop using it at all. Regardless of who was right, a usage policy contradicted by the use is not a control; it is a statement of intent, enforced, if at all, by contract law some months later.

”Meaningful human judgment”

The department’s answer to all of this is a good one: an official described the integration as “pairing access to powerful tools with clear safeguards, meaningful human judgment, and accountability for high-stakes decisions.” Those are the right words. The difficulty is that judgement in the loop is an assumption about behaviour under pressure.

In 2024 +972 Magazine, with Local Call, published an investigation by Yuval Abraham drawing on six Israeli intelligence officers involved in AI-assisted targeting. They described a system, Lavender, whose output was “considered accurate only 90 percent of the time” - one target in ten known in advance not to belong to the Hamas military wing at all. On the review meant to catch the difference, one officer said: “I would invest 20 seconds for each target at this stage, and do dozens of them every day.” Another: “we only checked that [the target] was a man - that was enough.” The IDF disputes the account, denying any such kill list exists and saying AI serves as an auxiliary tool, subject to independent examination by analysts as international law requires.

That is automation bias - what happens when a fast, confident, mostly-correct system is put in front of a person with more work than time. It requires nobody to be negligent.

The Pentagon has published nothing comparable about its own operations, and there is a related figure that must not be pressed into service here. In June its chief digital and AI officer, Cameron Stanley, filed a sworn declaration in a Mississippi lawsuit - a case about the gas turbines powering an xAI data centre - stating that the Grok Gov Model, running inside the Maven Smart System, “enabled US forces to deploy over 2,000 munitions to 2,000 distinct targets within 96 hours during Operation Epic Fury.”

That is a throughput figure. It says how much was done, not how long anyone spent deciding - and targets may be developed in parallel or planned far in advance. It would be easy, and wrong, to set it beside the twenty seconds above and let a reader draw the line: there is no public account of human review in that operation, and we do not know.

What it establishes is enough - the model chosen for its willingness to accept unrestricted lawful use was then used to flag targets in a war, a fact that surfaced only because a government lawyer had to explain to a court why a data centre’s electricity supply mattered.

Where a policy has to live to be a policy

None of this argues that model choice is unimportant - it argues that model choice is not a control.

A control is specific to your situation rather than averaged over everyone’s; it cannot be altered by the party it constrains; and it does not change without your knowing.

Safety expressed as model behaviour has none of those properties: calibrated by a vendor for a population that does not include your circumstances, negotiable though not by you, and moving whenever the model is updated - continuously, undisclosed, often unintended.

The alternative is not to switch safety off, nor to trust a better company. It is to notice that the context missing at the vendor sits with the organisation doing the work: what the system is for, who authorised it, what it may process, when that permission ends.

All of which points at where a policy has to live - a control point the operator owns, outside the model’s weights, because those are the vendor’s, and outside the prompt, because the person being governed can edit it. Somewhere that survives a model update, a vendor swap, a contract dispute and a change of administration, because it was never the vendor’s to remove.

Two limits. Nothing here would make a targeting decision safe, and nothing above should be read as suggesting otherwise: policy at the inference layer governs what a model may be asked and what it may answer, and has no standing in a rules-of-engagement question. What Maven and Lavender raise are questions of military process, law and accountability, and no middleware will settle them.

What it does address is duller and nearly universal - being able to say what your AI tools are permitted to do, in a form specific to you, that your users cannot renegotiate, and that does not quietly change when a supplier ships an update or loses an argument in Washington. Today that is mostly a paragraph in a wiki and a hope about the vendor’s roadmap.

We think it can be an instruction a machine enforces, rather than a quality a document claims, and that it belongs with the operator rather than the model. That is where our work is focused.

The Pentagon’s shortlist was decided by which supplier would agree to the fewest limits. Whatever you make of that as defence policy, it answers the question for the rest of us too. The safety policy your AI is following is its vendor’s - and a safeguard you did not build, on infrastructure you do not run, under terms you did not write, was never yours.

Sources

The deployment

The dispute and the criterion

The research

Model behaviour and automation bias