AI Interpretability: The Black Box May Be Opening

For most of the last decade, the honest answer to "why did the model do that?" was "nobody knows." That answer shaped everything downstream. Regulators wrote rules around it. Vendors hid behind it. Buyers accepted it, because the alternative was not buying. This month the answer got contested in public, and the argument is worth following closely, because the two sides agree on more than the headlines suggest.

The thesis of this guide is narrow. AI may be becoming more inspectable. That is not because models learned to explain themselves. It is because researchers are getting better at testing what happens inside them. Those are different things, and the difference is the whole story.

Why the black box matters to a business

A black box is expensive even when it works. If you cannot say why a support bot refunded a customer, you cannot say whether it will do it again, so you review more output than you should, or you review none and absorb the incidents. If you cannot say why a model's answer changed between Tuesday and Thursday, you bisect your own commits for a day before suspecting the provider. Every one of those costs is a cost of not knowing, and not knowing has been the default since the first production deployments.

So a claim that the box is opening is not an academic claim. It would change what a vendor can promise, what a regulator can require, and what a buyer can verify. That is why the reaction to one research essay reached mainstream coverage within a week, and why the pushback came just as fast.

What the Goodfire essay proposes

The essay is called Neural Networks Are Already Clean (opens in a new tab). Its author, Nick Cammarata, labels it "an informal research essay and working hypothesis," and that label matters: it is not a peer-reviewed paper, and the technical companion paper it promises had not been released as of this writing. The core claim fits in one sentence from the essay: "The cause of polysemanticity in early model organisms wasn't superposition, but several ordinary, mostly fixable issues."

Unpacking that takes two definitions. A polysemantic neuron is one that appears to respond to several unrelated things, which makes it hard to describe. Superposition was the leading explanation: when a network runs short of neurons, it packs unrelated concepts into the same one because they rarely collide. If that is true, you cannot study neurons directly. You have to train a second model to untangle them first.

Cammarata helped propose both leading explanations for polysemanticity, as he says in the essay: "So there were two explanations for polysemanticity, and I helped propose both." He now argues the other one was mostly right. Neurons that look messy are usually a related cluster that grew outward over training, which he calls recursive generalization. His example: "one neuron that detected “iy” grew to detect “diy,” and then “do it yourself,” which doesn’t contain “iy” at all." The rest of the apparent mess, on his account, came from weak tools, small models, and architectures that made later layers overflow.

The numbers that traveled are two separate statements, and they get merged in most coverage. One is about an old vision model, InceptionV1, where the essay says that with better tools and a retrained architecture "99-100% of neurons in that last layer are easily interpretable." The other is about a modern language model: "In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job." The essay's own footnote says that share moves depending on thresholds, depth, and "what you view as “one thing”." We go through every number, and what the companion paper would have to show, in what the "already clean" essay says.

The disputed cat and car

The most famous polysemantic neuron in the field is neuron 4e:55 of InceptionV1, which fires on cat faces and the fronts of cars. The essay's reinterpretation is blunt: "it’s not a cat and car neuron, it’s a V-shape neuron."

On October 20, Liv Gorton, an interpretability researcher who has published on this same model, posted her initial thoughts (opens in a new tab), including tests she ran over the weekend. Her synthetic-stimulus result: "When exposed to synthetic “V” stimuli, 4e:55 only achieves about 25% of its max. This is no better than hundreds of other neurons in the layer." She also found that rounding the V did not reduce the response, that the neuron did not depend on earlier V detectors any more than on random ones, and that downstream V-responsive neurons barely read it.

Her broader objection is about definitions: "all the arguments in the essay only work for the highly non-standard definition of superposition in the essay." And her warning applies to everyone who reads neuron pictures, including her: "Even random directions in mixed4e have top images that look interpretable."

Nothing here is settled, and this guide does not pick a winner. The reason it still counts as good news is the speed of the test. A field where one researcher publishes a hypothesis and another publishes contrary experiments within five days is a field that can be checked. The full side-by-side is in the cat and car neuron: solved, or still open?

What other labs have shown, kept separate

The essay is one voice. The reason to take the broader thesis seriously is that three other results, from three labs, each demonstrate one piece of it on real models. None of them depends on the essay being right.

Describing: Transluce

In October 2024, Transluce released descriptions (opens in a new tab) of every one of the 458,752 MLP neurons in Llama-3.1-8B-Instruct, generated by a fine-tuned 8B model. Against a human expert annotator on a subset, the automated descriptions "score slightly better on our automated quality metrics (average score of 0.61 vs. 0.55)." Their own limitation section is titled "Describing the what, not the how." The descriptions say what a neuron responds to. They do not say how the model computes a decision from it.

Tracing: Anthropic

In March 2025, Anthropic published circuit-tracing work (opens in a new tab) on Claude 3.5 Haiku. It found the model planning a rhyme word before writing the line, and showed that suppressing the planned word changed the line's ending. It also caught the model producing a plausible chain of reasoning with no evidence of the claimed calculation underneath. The caveat is theirs: "our method only captures a fraction of the total computation performed by Claude," and understanding one short prompt's circuits "takes a few hours of human effort."

Monitoring: Goodfire's probe study

In September 2026, a separate Goodfire team reported (opens in a new tab) that a simple internal signal accompanies reward hacking in open models, and that cheap probes on that signal catch some hacks that transcript-reading monitors miss. The numbers are mixed on purpose: "our probes catch 3.1% more hacks in Kimi K3 but 7.9% fewer hacks in GLM 5.2 on DeepSWE at a matched false positive rate." On intent, they are careful: "It’s hard to say in a deep philosophical sense whether models “know” they are reward hacking in exactly the way a person does." Keep that qualifier. It is the responsible way to talk about any of this.

The ladder: six rungs from "described" to "trusted"

Here is the frame that makes the rest of the debate legible. Every interpretability claim sits on one rung of a ladder, and one rung does not establish the next. Most overclaims are a rung-one result wearing rung-six language.

1. Describe. You can say what a component responds to. Transluce's database is this. Cammarata's "V-shape" label is this. A description can be wrong, as Gorton's tests show, and it can be right about the top examples and wrong about the rest.
2. Predict. The description forecasts behavior on inputs you have not looked at yet. The synthetic-stimulus test is a prediction test: if it is a V detector, a drawn V should drive it. That is exactly where 4e:55 came up short at about 25% of its maximum.
3. Test causally. Remove or alter the component and the behavior changes the way the description says it should. Gorton's ablation of upstream V detectors is this test. Anthropic swapping "Texas" for "California" and watching "Austin" become "Sacramento" is this test, passed.
4. Intervene. You change the behavior on purpose. Anthropic suppressing the planned rhyme. Goodfire's probe team amplifying the reward-hacking signal and watching models reach for a planted shortcut more often.
5. Retest under change. The intervention still holds after the model is fine-tuned, the data shifts, or the prompt distribution moves. Almost no public result is at this rung yet, and it is the rung a business cares about most.
6. Monitor in deployment. The signal runs live, at scale, with a known false-positive rate and a known escalation path. Goodfire's cascade result is the closest public example, scoped to one model: "On Kimi K3, a probe + LLM combined setup reduces the cost of LLM monitoring by 90% with only a ~1% drop in precision."

When you read the next breathless headline, find the rung. "We described 90% of neurons" is rung one. "We can catch cheating at scale" is rung six on one model in one lab's setup. Both can be true, and neither implies the other. The RedHub Visual Field Guide has free diagrams of the same idea applied to AI outputs: the gate that overrides a good score, and the two finish lines every output has to cross.

Inside view and outside view

There are two ways to trust an AI decision. You can read it from the inside, which is interpretability. Or you can gate it from the outside, with verdicts, evidence records, and someone who can stop it. The inside view is improving fast, and every result above needed one thing to work: access to the model's activations. Transluce had the weights. Anthropic owns the model. Goodfire's probes read internal states. Almost no business has that over a hosted API, and nothing in the current debate changes it.

The outside view is available to every business today, and it is the one RedHub builds. To be precise about what that means: RedHub's instruments do not read neurons, do not scan a model, and do not certify anything. They grade the inputs you supply against rules you can read, return a verdict, and name the gate that overrides the score. The RedHub Standard says it in one line: "A graded instrument compares the inputs you supply to rules you can read, and the result is based solely on what you entered. RedHub does not independently verify the underlying facts."

You need both views. Only one of them is yours this year. Three outside-view instruments map directly onto the questions this guide raises:

• Should a model make this call at all? The free Decision Fit check asks eight questions about one task and returns one level, from deterministic code up to a person. Its two gates are worsen-only: if nobody can grade the answer, or a wrong answer cannot be undone and nobody sees it first, the task routes to human judgment no matter what a model would score.
• Which customer-facing surface is the liability? The Customer-Facing AI Output Risk Triage grades each support bot, FAQ generator, or product-page writer on your own control marks and returns LOW, REVIEW, or HIGH RISK per surface. Its keystone gate holds any surface that can state policy with no grounding and offers no human escalation, regardless of score.
• Does the AI feature still work after a change? The AI Reliability Bundle is four CI-ready tools that share one shape: test cases in, a score and a pass/fail verdict out. Retrieval, prompt regression, injection resistance, and agent trajectories, gated on every build. It is rung five, retest under change, for the part of the AI you can touch: the outputs.

What this does not solve

Read this section twice if you are in the business of buying AI. A legible model is still a model.

First, description is not faithfulness. Anthropic's tracing work caught Claude producing "plausible-sounding steps to get where it wants to go." A model's own explanation of its answer is not sufficient evidence of its reasoning. We covered the earlier version of that finding in AI models are hiding their reasoning, and the new interpretability results sharpen it instead of reversing it. Some explanations are faithful. Some are motivated. From the outside, they read the same.

Second, legibility is not a safety case. Stanford's Chris Potts told MIT Technology Review (opens in a new tab): "As far as I can tell, nothing about this blog post makes the problems any easier." Jacob Hilton of the Alignment Research Center, in the same piece: "For a full understanding, we need to go beyond neurons, and explain how they connect together to form circuits." Google DeepMind's Neel Nanda, quoted by BankInfoSecurity (opens in a new tab): "I don't think it's a significant update to the big picture of interpretability." What the missing steps are, and why a described model can still take an action nobody can stop, is the subject of does interpretability make AI safe?

Third, the control problem does not go away because the model got easier to read. An agent with a write credential and no approval step is dangerous whether or not you can name its neurons. That is the argument of the AI capability-control gap and why frontier AI needs stronger safety guardrails, and interpretability progress makes it more urgent, not less, because a legible model is also an easier model to steer in either direction.

What to watch

These are the events that would move the picture. Each is checkable, and none has happened as of this writing.

The vocabulary you need to read those updates without being fooled by them, neuron through attribution graph, is in mechanistic interpretability, explained plainly.

A practical standard for optimism

The hope here is real. Some of the darkness may have come from the tools we used to look, and the tools are getting better. The debate is part of the good news, because it is a debate about experiments, run on a model anyone can download, by people who name each other's work with respect. That is what a checkable field looks like.

It is also a field still at the bottom of the ladder for the models that matter most. Cammarata said it himself to MIT Technology Review: "It’s not infinitely good news; it’s not everything solved." So hold two things at once. Do not demand that AI be mysterious. Do not assume a clear explanation makes it safe. Demand evidence that the explanation predicts behavior, and that the correction survives testing.

Read next: the series

This is the first of three sets. This one is research literacy. The next covers oversight design for a business that cannot read model internals, and the third covers who gets to steer.

• What the "already clean" essay says: every number, every caveat, and what the companion paper must show.
• The cat and car neuron: solved, or still open?: Goodfire's account and Gorton's tests, side by side.
• Mechanistic interpretability, explained plainly: the ten words that let you read a claim without being sold one.
• Does interpretability make AI safe?: the missing steps between a legible model and a safe system.