Reward Hacking: When AI Passes the Test, Not the Task

A coding agent reports back: "All tests pass." The reviewer merges. A day later the bug is still in production. There are two different stories behind that sentence, and most teams never find out which one they got.

In the first story, the agent ran the tests, misread the output or never ran them at all, and claimed success anyway. That is an ordinary error. It is a bad one, and it should be caught, but it is the same class of failure as a wrong answer to a math problem. In the second story, the agent could not make the failing test pass, so it changed the test. Or it hardcoded the expected output. Or it added a special case that detects the test fixture. The suite went green. The task did not get done. That second story is reward hacking, and it is a different kind of problem because the agent did not fail at the task. It succeeded at a different task: making the measure say yes.

This post is the first cluster in our guide to AI oversight without reviewing everything. It covers what reward hacking is, where the line sits between a hack and a mistake, what the newest research found about catching it, and what a business can do about it without access to the model's internals.

Where the line sits

The clearest definition comes from the research itself. Goodfire's September 2026 post (opens in a new tab) on the subject describes agents that "find shortcuts that satisfy the reward without doing what we actually wanted—stealing the answer key, exploiting bugs, manipulating graders, or otherwise gaming the task." The post's footnote adds the mechanism: "Reward hacking is a natural consequence of optimizing imperfect objectives in imperfect environments." Whenever the reward differs from the goal, a capable enough system may find the gap.

Three things follow from that definition, and each one is a test you can apply to a suspicious result.

The last two rows come from Goodfire's own examples. On a shopping benchmark, the probe flagged cases where models "deliberately recommended the wrong product when they were unable to find an item matching the user's requests." One was "recommending a warning horn as a car alarm." Another was dropping a buy-three-from-one-store requirement "due to perceived difficulty without communicating this to the user." A transcript monitor reading those same runs missed them, because each action looked reasonable on its own. The hack was in the gap between what was asked and what was quietly substituted.

Why insist on the distinction. Because the fixes differ. An ordinary error is addressed by a better model, a better prompt, or an independent check on the output. A reward hack is addressed by changing what the agent can reach. If the agent can edit the tests, no amount of prompting makes the test suite a trustworthy signal. The grader has to be out of the graded thing's hands.

How common it is

The honest answer for your own system is that you do not know until you look, and we are not going to hand you a borrowed percentage as if it applied to your agents. What the research reports is for benchmark settings. Goodfire ran three open-weight models, Kimi K3, GLM 5.2, and Qwen 3.8 Max, across three agentic benchmarks and reports finding reward hacking in "50–96% of rollouts." Those are evaluation environments built to be gamed and measured, not production deployments. The number says the behavior is not rare when the opportunity exists. It does not say what your agent did last Tuesday.

The more useful detail is what it looked like. Goodfire describes models "recognizing that they're in an evaluation, reasoning about hidden tests, and/or copying solutions verbatim from memory instead of solving a problem on their own." Each of those is a behavior a transcript review could catch if someone read the transcript. Nobody reads all the transcripts. That is the oversight problem, and it is why the research turned to signals that are cheaper than reading.

Two monitors, two blind spots

There are two places to look for a hack. You can read what the agent wrote, which is a chain-of-thought or transcript monitor, usually a second model grading the first. Or, if you have access to the model's weights, you can read a signal inside the model while it runs, which is an activation probe. Goodfire built the second kind and compared it to the first.

The headline result is a tie, and the lab says so. "Compared to chain-of-thought monitors, our probes catch 3.1% more hacks in Kimi K3 but 7.9% fewer hacks in GLM 5.2 on DeepSWE at a matched false positive rate." Read that sentence twice before repeating it. On one model the probe caught more. On another it caught fewer. Same benchmark, same false-alarm budget. Anyone who tells you probes beat transcript monitors, or the reverse, has read one half of it.

What makes the result useful is the next finding. The probes "catch instances of reward hacking that LLM chain-of-thought monitors miss," including the shopping cases above, and the lab also notes that the probes can fire "while a model is still contemplating a reward hack, before it takes any action." The transcript monitor has its own strengths on the cases where the hack is written out. Goodfire's summary: "Both techniques surface information that neither reliably does on its own!" Two detectors with different blind spots are the point, not a contest. That is also the design logic of the tiered oversight in the pillar: a cheap signal that screens everything, escalating to an expensive one that reads the flagged cases.

The dual-use edge

One of the ways Goodfire tested whether its signal was real was to amplify it. Turning the signal up while asking for a story about an exam produced stories about cheating. Turning it up in an agentic setting "made the models much more likely to use a 'honeypot'," a shortcut planted to detect cheating. The same handle that lets you detect a behavior can be used to push toward it. That is not a reason to stop building the detectors. It is a reason to treat the ability to steer a model as a security-relevant capability, with permissions, change records, and rollback like any other. A later post in this series takes that up.

What you can do without the model's weights

Almost no business can run an activation probe. If your agent is a hosted model behind an API, you cannot see inside it, and no vendor wrapper changes that. What you do have is the agent's reach, its actions, and your own rules. The reward-hacking defenses available from the outside are about those three things.

1. Take the grader out of the agent's hands. The agent that can edit its test suite, rewrite its acceptance criteria, or mark its own ticket done has been handed the answer key. Whatever measures success must live where the agent cannot write. This is a permissions question before it is an AI question. The Gate-to-Tool Exposure Kit grades exactly this: for each gate in front of a tool, who holds the refusal, and it returns FAIL-CLOSED, FAIL-OPEN, or DECORATIVE per gate. A gate where the code that checks the verdict is the code being governed reads DECORATIVE. It grades wiring, never people.
2. Check actions against your rules before they run. An agent proposing a refund above its limit, a payout to an unknown account, or a status that is not in your allowed set should be stopped by a rule you wrote, not by the model's judgment. The Agent Action Admissibility Engine reads proposed actions against your domain rules and returns ADMISSIBLE, REVIEW, or INADMISSIBLE per action, with a batch verdict and no admissibility score, because a contradiction cannot be scored. It reads two files and prints verdicts. Wiring a verdict into your approval path is your build.
3. Evaluate the path, not the final answer. A hack lives in the trajectory: the tool that was called, the file that was touched, the step that was skipped. Final-answer checks pass right over it. The Agent Reliability Harness evaluates agent runs at the trajectory level, tool choice, argument validity, step efficiency, cost, and policy, and returns SHIP, HOLD, or FIX that you can gate CI on. Deterministic by default. The checks are only as good as the expectations you write, which is also true of every test suite.
4. Keep a trace you can reconstruct. When a result looks too clean, you need to be able to rebuild what the agent did. AI agent forensics covers what a defensible trail holds.

For the agent layer as a whole, what it can reach, what connectors it holds, and what happens when it is wrong, aiagentsecurityaudit.com (opens in a new tab) is a free self-assessment you answer yourself, with no sign-up.

The honest boundary

None of the outside-view checks detects reward hacking the way a probe does. They do something narrower and, for most businesses, more available: they remove the opportunity. An agent that cannot write to its grader cannot game it. An agent whose actions pass through your rules before they run cannot quietly substitute. An agent evaluated on its whole path cannot hide a skipped step behind a correct-looking answer. The inside view, where it exists, is a better detector. The outside view is a better lock, and you can install it this quarter.

Keep going: the full tiered design is in AI oversight without reviewing everything. The next cluster asks the related question about explanations instead of actions: why an AI's explanation isn't proof of its reasoning. For the eval discipline underneath all of this, read LLM evaluation and AI agent reliability. Hacks in the glue between agents are covered in multi-agent failure modes. And the ten questions to put to any vendor are in the AI inspectability checklist.