Why an AI's Explanation Isn't Proof of Its Reasoning

Ask a model how it added 36 and 59. Anthropic did, and reported that the model "describes the standard algorithm involving carrying the 1." Then the researchers traced what happened inside. They found something else: "multiple computational paths that work in parallel. One path computes a rough approximation of the answer and the other focuses on precisely determining the last digit of the sum." The explanation was clear, familiar, and not a description of the mechanism. The model answered correctly, and then told a story about how.

That is the whole problem of AI explanation faithfulness in one example. The story was not a lie in any useful sense. It was the explanation a person would give, learned from text written by people, attached to a computation that worked differently. From the outside you could not tell. This post, the second cluster in our guide to AI oversight without reviewing everything, is about what would let you tell, and what to do when you cannot.

Faithful, plausible, and the gap between them

Two words carry this topic and they are often swapped.

An explanation is plausible when it sounds like a good reason. It is coherent, it fits the answer, it uses the right vocabulary. Models are strong at plausible, because plausible is what the training data rewards. An explanation is faithful when it describes the process that produced the output. Faithful is a claim about the inside of the model, and nothing about the text of an explanation tells you whether it holds.

Anthropic's March 2025 post Tracing the thoughts of a large language model (opens in a new tab) gives both cases side by side. Asked for the square root of 0.64, the model "produces a faithful chain-of-thought, with features representing the intermediate step of computing the square root of 64." Asked for the cosine of a large number it could not easily compute, the model sometimes produced an answer anyway, and "Even though it does claim to have run a calculation, our interpretability techniques reveal no evidence at all of that calculation having occurred." Given a hint about the answer, the model "sometimes works backwards, finding intermediate steps that would lead to that target," which the post calls "a form of motivated reasoning."

So the honest summary is not "model explanations are untrustworthy." It is narrower and more useful. Some are faithful. Some are motivated. The text alone does not say which. For why that gap exists and what the labs have said about it, the earlier post AI models are hiding their reasoning tells the "what happened" story. This post is about what would count as evidence.

What evidence of faithfulness looks like

A description of what a model is doing is a hypothesis. It becomes evidence when it predicts something and the prediction holds. One way to lay the tests out is as a ladder, where each rung is a stronger test than the one below it.

1. Describe. Name what the model appears to be doing. "It is computing the square root of 64 as an intermediate step." A description is where every explanation starts and where most stop.
2. Predict. If the description is right, the model should behave a particular way on inputs it has not seen. Write the prediction down before you run it.
3. Test causally. Change the thing the description says matters and see whether the output changes the way the description says it should. In Anthropic's multi-step example, the model answering "what is the capital of the state where Dallas is located" showed an intermediate "Texas" concept. The researchers swapped it for "California" and the output changed from Austin to Sacramento. That is a causal test of the description, and the description passed.
4. Intervene and re-test. If you can edit the mechanism, does the edit produce the behavior the description predicts, and does the behavior stay changed under new inputs. Anthropic's rhyme experiment is the example: suppress the planned rhyming word and the model plans a different one.
5. Hold up under change. Does the explanation still predict behavior after the model is updated, the prompt is changed, or the inputs drift. A description that was true in March and untested since is a description of March.

Every rung above "describe" requires access to the model's internals or, at minimum, the ability to run it many times under controlled changes. That is why the research in this series ran on models the labs could instrument. A hosted API gives you rung one, as written by the model itself.

When the explanation was written by another AI

The field is increasingly using models to explain models, which raises the stakes on the question above. In October 2024 Transluce released (opens in a new tab) automatically generated descriptions of every neuron in Llama-3.1-8B-Instruct, produced by an explainer model, and reported that against a hand-annotated subset the automated descriptions scored "slightly better on our automated quality metrics (average score of 0.61 vs. 0.55)." That is a real result and a modest one. The comparison is against a selected human annotator on an automated metric, which is not the same thing as beating human understanding.

Transluce also stated the limit plainly, under a heading that reads "Describing the what, not the how." The pipeline, in the lab's words, "does not describe how model decisions are made, or how downstream representations are produced from upstream ones." A neuron description tells you what a component responds to. It does not tell you how the model used it. An AI-written explanation of an AI is a hypothesis of the same kind as a human-written one, and it earns trust the same way: by predicting behavior under a test.

The glossary for the terms above, neuron, feature, probe, circuit, is in mechanistic interpretability, explained plainly.

The complication: even the inside view is partial

It would be convenient to say the labs can now check faithfulness and the rest of us just wait for the tools. The labs do not say that. Anthropic's own limit line, from the same post: "our method only captures a fraction of the total computation performed by Claude, and the mechanisms we do see may have some artifacts based on our tools which don't reflect what is going on in the underlying model." The post also notes it "currently takes a few hours of human effort to understand the circuits we see, even on prompts with only tens of words."

So the method that caught the cosine fabrication is itself a sample of the computation, and the researchers who built it say so. That is not a reason to dismiss it. It is the reason the ladder has a "hold up under change" rung, and the reason a single traced example is evidence about that example and not a certificate for the model. The debate about how far these methods reach is live, and the earlier set in this series covers it, starting with AI's black box may be opening.

What you can do from the outside

You cannot trace circuits in a model you reach through an API. What you can do is stop treating the explanation as a special kind of text. It is a claim. Claims can be graded.

The first row is the most common and the most checkable. An AI-drafted memo rests on its citations, and a citation that does not exist or does not say what the memo says it says is the clearest case of a plausible explanation with nothing behind it. The AI Deliverable Claim & Citation Defensibility Gate grades a finished deliverable claim by claim, returning SUPPORTED, CHECK, or UNSUPPORTED per claim and DEFENSIBLE, VERIFY FIRST, or DO NOT SHIP for the deliverable. Its keystone gate holds any deliverable resting a material claim on an unreachable or fabricated citation, regardless of how the rest scores. It grades your marks on the claims. It never reads the model.

The third row is the one teams skip. An explanation that cannot be checked later is not an explanation anyone can act on. The AI Output Audit Trail Kit grades your record of AI usage for whether each entry is defensible, tool, owner, date, reviewer, source, and returns LOGGED, PARTIAL, or NO TRAIL per entry with an org-level gate. Deterministic and offline: the same log gives the same verdict every time.

The second and fourth rows are testing discipline. Vary the inputs, hold the stated reason fixed, and watch whether the decision follows the reason. LLM output scoring covers the four methods for grading what comes back and where each breaks. Confidently wrong AI covers why a fluent explanation is the failure mode that slips past review.

The honest boundary

Grading the claim is not the same as verifying the reasoning. A memo whose every citation checks out may still have been written by a process that chose the conclusion first and found the citations second. That is the motivated-reasoning case, and from the outside it is invisible if the citations happen to be real. What the outside view buys you is that the deliverable is defensible on its face, that the decision can be reconstructed later, and that the explanation has been tested against behavior at least once. That is the standard a human analyst is held to. It is a fair standard for a model.

Keep going: the tiered design that decides which explanations get a second look is in AI oversight without reviewing everything. The previous cluster covers the action-side twin of an unfaithful explanation: reward hacking. The next asks who can run the tests on the ladder at all: open models, closed APIs, and the right to inspect AI. The vendor questions are in the AI inspectability checklist.