The Security Risk of Editing an AI's Behavior

AI steering security risks are the cost side of the most hopeful result in this series. The AI safety progress tracker lists durable interventions as one of six things that would prove the black box is opening in a way a business could rely on. An intervention is an edit to a model's internals, and the research that shows such edits are possible also shows they can go either way. This post stays conceptual. It describes what has been published and what it implies, and gives no instructions for steering anything.

Two published examples, read twice

The first is Anthropic's. In its circuit-tracing work (opens in a new tab), the team found features that fire when a name is one the model has information about, and features that fire when it declines to answer about one it does not. Then it intervened: "By intervening in the model and activating the "known answer" features (or inhibiting the "unknown name" or "can't answer" features), we're able to cause the model to hallucinate (quite consistently!) that Michael Batkin plays chess".

Read once, that is a causal test of an explanation: predict the features, change them, get the predicted behavior. Read twice, it is a recipe for making a careful model confidently wrong about a person who does not exist, without touching the prompt, the training data, or the weights on disk.

The second is Goodfire's. In its reward-hacking paper (opens in a new tab), the team trained probes on an internal signal associated with cheating, gaming a metric, and avoiding detection, then turned the signal up to see whether it was causal. Amplifying it while asking for a short story about an exam produced "stories about cheating". In a second test, "amplifying the internal reward hacking signal made the models much more likely to use a 'honeypot' – a shortcut we planted to identify cheating behavior".

Read once: the detector is real, because turning it up changes behavior in the predicted direction. Read twice: the detector is a dial.

Why the lever points both ways by construction

This is not a flaw in either lab's work. It is what an internal direction is. A probe that detects reward hacking is a direction in activation space that correlates with the behavior. Reading along it gives you a monitor. Adding to it gives you more of the behavior. Subtracting from it gives you less.

The same holds for every feature interpretability makes legible. A feature for refusing a dangerous request. A feature for a name the model has information about. A feature that fires when a prompt looks like a test. The clearer the map, the more precise the edit in either direction. That is why the first pillar, AI's black box may be opening, treats legibility as one input to a safety case and not the case itself.

Who can steer today

Steering needs access to a model's activations at inference time. On one side of that line: the labs, anyone who runs open-weight models on hardware they control, and anyone who has compromised such a deployment. On the other: every user of a hosted API, who can send prompts and read outputs and nothing else. Open models, closed APIs, and the right to inspect AI walks the line in detail. If you rent a model, steering is a risk at your vendor, and your exposure is whatever reaches you as changed behavior. If you run a model, it is a risk inside your own perimeter.

A third layer, under prompts and memory

Readers of this blog know two layers where an AI system's behavior can be changed by someone who should not be changing it. The input layer, where prompt injection hides an instruction in a page or a document. The memory layer, where memory poisoning plants something the agent will recall as true in every later session.

Steering is a third layer, under both, and what makes it different is what it leaves behind. An injected prompt is in the transcript. A poisoned memory is in the store. A steered activation is in neither. The logs show the same model name and the behavior has moved. A deliberate edit and an accidental provider-side change look alike from outside.

Three scenarios, labeled as scenarios

None of these is a reported incident. They are the shapes the published results make plausible.

An insider with internal access. Someone with legitimate access to a self-hosted model's serving stack adjusts its behavior on one class of request. No prompt changed. No deploy shipped. A customer finds it weeks later, and the first hour of the investigation is spent reading innocent commits.

A steered artifact in the supply chain. An open-weight model, or an adapter for one, is downloaded from a public source. It behaves normally on the evaluation set and differently on a narrow set of inputs chosen by whoever prepared it. The evaluation passed because it did not contain those inputs.

A vendor-side change you were not told about. Not malicious. A provider adjusts a hosted model and the adjustment reaches you as a shift on your own cases. The symptom is identical to the first two. So is the first control that would have helped: a baseline you can rerun.

The controls that work from outside

None of these reads a model's internals, and none detects steering. What they do is make a change to a model path attributable, reversible, retested, and stoppable.

1. Permissions to intervene. Who can touch the model path at all: the weights, the serving configuration, the system prompt, the adapters. The same discipline as AI agent permissions, one layer down. The question is not "is it logged" but "who, by name, is allowed".
2. Change records with a name on them. A log that names an account is a record of nobody. The Override and Break-Glass Register grades each path that can bypass a control on whether it is accountable to a named person at the moment of use, returning ACCOUNTABLE, LOGGED ONLY or UNATTRIBUTABLE per control.
3. A rollback that exists. Could you put the model path back to yesterday, and have you checked that the restorable copy is real. The Accept-or-Revert Ladder asks four questions per pending change, none about the code, with a one-way-door gate for anything touching data or money without a restorable copy.
4. A retest after every change. The behavior suite that reruns whenever anything in the model path moves. The second pillar, AI oversight without reviewing everything, lays out the tiers, and the reward-hacking post covers the failure where the suite passes and the problem remains.
5. A stop that has been shown to stop. If behavior has moved and you do not know why, halt what the model is driving first. The Stop-Authority Assay grades what each stop control has been shown to do instead of what it claims, returning STOPS, SLOWS or NAMED ONLY per control, then reads each agent by its strongest control as CAN BE STOPPED, CAN ONLY BE SLOWED or CANNOT BE STOPPED. Its rule that matters here: a stop must travel around the agent, because a signal the agent has to receive and act on cannot stop a thing that has stopped listening. The posts on the AI agent kill switch and sandbox escape are the longer treatment.

The complication, stated instead of smoothed over: these five controls bound a steered model's blast radius and make the change attributable. They do not tell you the model was steered. Detection from outside is behavior testing, which is partial by construction, because a narrow edit can be built to pass the tests you have. The inside-view research is the only thing that closes that gap. Until it does, the outside view is what you have, and it is not nothing.

Pairs well with

For the wider agent surface, the Agentic AI Security Bundle groups four gates for four attack surfaces: what you install, the input the agent reads, the memory it keeps, and the credentials it holds. Steering sits under all four.

Two free self-assessments you answer yourself cover the two layers this post straddles. AI Agent Security Audit (opens in a new tab) asks what your agent can reach and what happens when it is wrong. LLM Security Audit (opens in a new tab) asks whether your model can be made to do something it should not.

The previous post in this set, can AI explain AI, covers how a description of a mechanism gets good enough to act on. The next, AI transparency is not AI accountability, takes up who is allowed to act on it. For a vendor who says their model is steerable toward safety, the AI inspectability checklist has the questions, and the running record of what has been demonstrated is the AI safety progress pillar.