Can AI Explain AI? Trusting Machine-Written Explanations

Can AI explain AI? The question used to be rhetorical. It is now a working method. Transluce's description pipeline, Anthropic's circuit tracing, and Goodfire's October essay all lean on a model to describe what another model is doing, and the Goodfire essay in particular reads as a collaboration between a researcher and agents. It belongs in the AI safety progress tracker because it is both the most hopeful development and the easiest to overread. If more capable AI helps build the tools that inspect AI, the inspection scales. If the tools write convincing descriptions of nothing, the illusion scales too.

What an AI-written explanation is

In Transluce's pipeline (opens in a new tab), published in October 2024, an explainer model is shown the inputs that make a neuron fire most strongly, and the inputs that make it fire least. It is "prompted to make hypotheses that explain the data, and hypotheses are evaluated using an automated scoring procedure". The explainer writes several candidate descriptions in plain language. A second model, the simulator, reads each description and predicts how the neuron would fire on held-out text. The description whose predictions best match the real activations wins. Transluce's own word for each candidate is the right one: hypothesis.

Goodfire's essay describes a tighter loop, which it calls blind maximum: "One agent investigates a neuron and writes a short explanation", and a second agent, in the essay's words, "sees only that explanation and writes an example meant to activate it maximally". If the second agent's example fires the neuron, the explanation carried enough information to predict behavior. A separate check in the essay is larger. For 200 random neurons, an agent read only each explanation and wrote five new examples and five near misses before any were measured. For the 183 whose explanations describe their positive side, the new examples fired their neurons strongly 76% of the time, against 11% for the near misses.

Those are two different tests, and they test different things. Transluce's scores how well a description simulates activations on text the explainer never saw. Goodfire's scores whether a description is specific enough that a fresh reader can trigger the neuron. Both are prediction tests. Neither is a causal one.

What 0.61 against 0.55 means

The number that travels is Transluce's: automated descriptions "score slightly better on our automated quality metrics (average score of 0.61 vs. 0.55)" than descriptions written by a human expert. It deserves its exact framing.

The comparison was against one selected expert annotator, who spent around 3.6 minutes per neuron. The metric is Transluce's own simulation score, not a human judgment of whether the description is true. And Transluce flags the sampling: the descriptions "were scored on mostly top-activating exemplars which means that they are not representative" of the neuron's behavior across the full range of inputs. A description tuned to the loudest examples can be right about those and wrong about the quiet ones.

Transluce's limitations section carries a heading worth memorizing: "Describing the what, not the how". The team writes that its descriptions "could be useful inputs to procedures that produce causal descriptions of model behavior, but do not themselves provide an algorithmic description" of the computation inside the model. That is the lab saying, in its own words, that the output is a hypothesis for the next test.

Interpretability illusions

The reason an AI-written description cannot be trusted on fluency alone has a name in the literature. Liv Gorton, in her response to the Goodfire essay (opens in a new tab), points to Bolukbasi and colleagues' 2021 work on "interpretability illusions", in her summary cases "where neurons and random directions looked interpretable on one dataset and told a different story on another."

Then she shows it. She pulls the top images for six random directions in the same layer the essay studies, random combinations of the layer's 832 neurons, not cherry-picked, "just the first 6". They look like they mean something. Her line is the one to keep: "Even random directions in mixed4e have top images that look interpretable."

Her test of the essay's flagship example makes the same point from the other side. The essay reads neuron 4e:55 as a V-shape detector. Gorton found that on synthetic V stimuli the neuron "only achieves about 25% of its max", and that the neurons which do respond strongly to V shapes are different ones. Whether she or the essay is right about 4e:55 is open, and the cat-and-car post lays both cases out without picking. The point for this post is narrower. A description that fit the top examples did not predict the response to a controlled stimulus, and nobody could have told that from reading the description.

Now imagine the description was written by an agent, in confident prose, at scale, for ninety percent of the neurons in a model. The illusion does not get smaller because the author is a machine. It gets faster.

The ladder a description has to climb

The first pillar of this series uses a ladder for reading any interpretability claim: describe, predict, test causally, intervene, retest under change, monitor in deployment. An AI-written explanation sits on rung one. Here is what each next rung asks of it.

1. Describe. The explainer writes a hypothesis from examples. This is where Transluce's and Goodfire's pipelines produce their output. Fluency is free at this rung.
2. Predict. The description is used to guess behavior on inputs nobody chose: held-out text, a synthetic stimulus, an example a second agent writes blind. Both labs run a version of this. It is the first rung where a description can fail, and Gorton's 25% is what failure looks like.
3. Test causally. Change the thing the description names and see whether the behavior changes. Anthropic's circuit-tracing work has the clearest published example: by activating the model's "known answer" features and inhibiting its "can't answer" features, the team was able to make the model hallucinate, "quite consistently", that a made-up person plays chess. A causal result, which Anthropic bounds in the same paper: "our method only captures a fraction of the total computation performed by Claude".
4. Intervene, retest, monitor. Fix a behavior through the mechanism, check that the fix holds after the model changes, and watch it in production. No AI-written description has been shown to carry a model through these rungs yet. The status table in the pillar tracks whether that changes.

A description that stops at rung one is a label. The labs know this, and say so. The risk is in the retelling, where "an agent explained 90% of the neurons" drops the word "hypothesis" on the way to the headline.

The same shape, outside the lab

The structure of "AI explains AI" is the structure of every AI-written explanation you read at work. When a model tells you why it recommended a vendor, flagged a transaction, or ranked a candidate, it is producing a description after the fact, from the same machinery that produced the decision. Anthropic's tracing work found that when given a hint about an answer, the model "sometimes works backwards, finding intermediate steps that would lead to that target", which the paper calls a form of motivated reasoning. Some explanations are faithful. Some are not. The text does not tell you which. Models hiding their reasoning covers what happened, and why an AI's explanation is not proof of its reasoning covers what evidence of faithfulness looks like.

So the rule is the same at both scales. An explanation, whoever wrote it, is a claim about a mechanism. You test it against something the explanation did not produce: the record of what the system did, a controlled input, a second reviewer who did not see the explanation first. The confidently wrong failure is what happens when that test is skipped because the prose was good.

What to do with an AI-written explanation

Separate the explanation from the evidence. Write down what it claims, then what you can check independently. If the second list is empty, file the explanation as a hypothesis.

Look for the quiet examples. Transluce's descriptions were scored on the loudest activations. An explanation of a business decision is usually built the same way, from the clearest cases. Ask what it says about the borderline case, then go find one.

Grade the reader, not just the text. The weak point in "AI explains AI" is a human who accepts the description because it is fluent. That habit is trainable, and it is the one the research cannot fix for you.

Pairs well with: the outside view

Nothing RedHub sells reads a model's internals. The RedHub standard says a result is based solely on the inputs you supply, with no independent verification of the underlying facts. The instruments below grade the human side of the explanation problem.

• The AI Over-Reliance and Critical-Thinking Check seeds AI text with a fabricated statistic, plausible-but-false claims, and biased framing, then scores how many a reviewer catches: INDEPENDENT, ASSISTED or RELIANT, with a hard gate on the fabricated statistic and an over-confidence flag. It is for individual development, not employment decisions.
• The AI Deliverable Claim and Citation Defensibility Gate grades a finished AI-assisted deliverable claim by claim, returning DEFENSIBLE, VERIFY FIRST or DO NOT SHIP, with a keystone gate that holds any deliverable resting a material claim on an unreachable or fabricated citation. It is the same test you would want to run on an explanation: is the support there, and does it support this.

For the vendor conversation, the AI inspectability checklist has the ten questions to ask anyone who says their product is explainable. The next post in this set, the security risk of editing an AI's behavior, follows the ladder one rung further, to what happens when the description is good enough to act on. The whole set sits under the AI safety progress pillar.