Mechanistic Interpretability, Explained in Plain Terms

Mechanistic interpretability is the attempt to explain what an AI model does in terms of its internal parts and how they connect, the way you would explain a watch by its gears instead of by asking it the time. Every headline in the current debate about whether AI's black box is opening is built from about ten technical words, and most overclaims come from one of those words being quietly swapped for a bigger one. This post defines the ten, with an example from the primary research for each, so you can read the next claim without being sold one.

TL;DR: Neuron, pole, feature, polysemanticity, superposition, circuit, sparse autoencoder, probe, steering, attribution graph. "Roughly 90% of neuron poles have one main job," a preliminary estimate, is a claim about poles, which is a claim about description, which is not a claim about circuits, prediction, or safety. Two of the ten words, feature and superposition, are currently used with different meanings by different researchers, and that difference is most of what the Goodfire-Gorton dispute is about.

Why the vocabulary matters

Here is the swap in one example. Goodfire's essay says that in preliminary audits of Qwen3-32B "roughly 90% of neuron poles have one main job." Retell that as 90% of the model being understood and you have changed the claim in three places. A pole is half a neuron. "One main job" is a description. "Understood" implies you can predict and change what the model does. Knowing the words is what lets you see the gap, and the gap is where the risk lives for anyone deciding what to trust.

The ten terms

1. Neuron

A single unit inside a network that takes in numbers from the units before it, combines them with learned weights, and passes a number forward. In a language model the neurons people study most sit in the feed-forward blocks, and there are a lot of them: Transluce's 2024 project described all 458,752 of them in Llama-3.1-8B-Instruct. The essay's reframe is worth keeping: "A neuron is less about what it sees than about what it does to the flow coming in, and often its job is to gate a region of the network." A neuron is not a concept. It is a switch that may or may not line up with one.

2. Pole

A neuron's output can run positive or negative, and the two directions can mean different things. The essay describes a neuron that "separates two subtle patterns at its positive and negative poles (a V versus an O)." That is why the 90% figure is stated in poles, not neurons: each neuron is being read as two possible jobs, one per direction. Transluce builds separate example sets for the most-positive and most-negative activations for the same reason.

3. Feature

A pattern in the model's internal activity that corresponds to something a human can name. The careful definitions are deliberately broad. Transluce: "a feature is any function from its representation to the real numbers." A feature may be a single neuron, or it may be a direction that mixes many neurons. Gorton, describing how the field's own notes use the word, calls features "independent directions that can’t be broken down further into meaningful parts." The essay prefers to study mechanisms like neurons instead of hypothesized features, and says why: "I find hypothesized features hard to think with." That preference is a methodological choice, and both camps know it.

4. Polysemanticity

The essay's definition: "the broad name for a neuron that appears to have multiple unrelated jobs or that looks impossible to understand." The famous example fires on cat faces and car fronts. The key word is "appears." As the essay puts it, "Polysemanticity is often more a claim about the observer than the neuron." A neuron can look polysemantic because of weak tools, too little context, a pattern nobody recognized, or because it mixes unrelated things. Telling those apart is the work, and it is where the cat and car neuron is currently stuck.

5. Superposition

This is the contested word, and the honest definition is two definitions. The field's standard sense, which Gorton quotes from the 2020 "Zoom In" article, is that polysemantic neurons "result from a phenomenon we call ‘superposition’ where a circuit spreads a feature across many neurons, presumably to pack more features into the limited number of neurons it has available." Later work formalized that as more features than dimensions. The essay's sense is narrower: a learning principle where "the network packs unrelated things into the same neurons because they rarely collide."

Gorton's central objection is that "all the arguments in the essay only work for the highly non-standard definition of superposition in the essay." Cammarata, in a clarification she published, now thinks he should have coined new terms. Until that is sorted out, any sentence of the form "superposition is wrong" needs a follow-up question: in which sense. Nothing in the current evidence supports a sentence declaring superposition dead, and this guide does not write one.

6. Circuit

A set of components wired together to perform a recognizable computation. Anthropic's 2025 tracing work gives the cleanest small example: asked for the capital of the state containing Dallas, Claude activates a representation of Texas and then of Austin, and swapping the Texas step for California changes the answer to Sacramento. That is a circuit, tested causally. Jacob Hilton of the Alignment Research Center, in posts on X quoted by MIT Technology Review (opens in a new tab), put the hard part here: "For a full understanding, we need to go beyond neurons, and explain how they connect together to form circuits."

7. Sparse autoencoder (SAE)

A second, smaller model trained to rewrite a layer's activity as a larger set of features, each of which is active only rarely. If neurons are tangled, an SAE is the untangling tool, which is why most recent interpretability work runs through one. The two sides read SAE results differently. The essay trained SAEs on its own vision models and reports that on the last layer "88% of its activity is near-copies of single neurons," and that what the SAE added was "loose groupings built from existing neurons." Gorton's earlier paper found curve detectors in InceptionV1 that the ten known curve neurons miss, which she reads as evidence that SAEs find real structure neurons alone do not. Both results can be true of different layers and different questions.

8. Probe

A small detector trained on a model's internal activations to flag when a concept is present. Goodfire's September 2026 reward-hacking study describes them as "small, purpose-built detectors that use a live “brain scan” of a model." Their probes were simple difference-of-means directions, and the team's own numbers show the tradeoff: against transcript-reading monitors, "our probes catch 3.1% more hacks in Kimi K3 but 7.9% fewer hacks in GLM 5.2 on DeepSWE at a matched false positive rate." A probe needs access to activations. If you call a model over an API, you cannot run one.

9. Steering

Changing a model's behavior by pushing on an internal direction instead of changing the prompt. Three examples from the sources. The essay's Colorado neuron: "Steering it made the model want to go hiking." Anthropic injecting the concept "green" into a poem and getting a line that ends in "green." Goodfire's probe team amplifying the reward-hacking signal and watching models use a planted shortcut more often. Steering is the clearest proof that a description is causal. It is also the clearest reason a legible model is not automatically a safer one, because the same handle turns both ways.

10. Attribution graph

A map of which components contributed how much to a particular output, traced backward from the answer. Anthropic's circuit-tracing method produces these, and their caveat applies to all of them: "our method only captures a fraction of the total computation performed by Claude," and the mechanisms seen "may have some artifacts based on our tools." The essay reports agents tracking "hundreds of thousands of attribution graphs" on its vision models. An attribution graph is evidence about one input. It is not a general account of the model.

Two approaches, one table

With the words in place, the methodological split in the current debate is easy to state.

Neither column is the fringe. Google DeepMind's Neel Nanda, quoted by BankInfoSecurity (opens in a new tab), said "I don't think it's a significant update to the big picture of interpretability," while describing neuron-level work as a reasonable approach with some advantages over the standard alternatives. The essay's own minimum claim is that "a network’s own mechanisms should be part of the toolkit." The two columns may end up as tools in the same kit.

How to read a claim with these words

1. Find the unit. Neuron, pole, feature, or circuit. A percentage of poles is not a percentage of behaviors.
2. Find the verb. Described, predicted, ablated, steered, or monitored. Each one is a different rung, and the pillar's six-rung ladder places them.
3. Find the model. A 2014 vision network, an 8B open model, or a frontier model behind an API. Results do not transfer across those by default.
4. Find who can check it. If the model and data are public, someone can reproduce it. If not, as Gorton puts it, "the same method has to be taken on trust."
5. Find the definition. If the word is "superposition" or "feature," ask which sense before agreeing or disagreeing.

The outside view: you do not need these words to check an output

Everything above is the inside view. It requires weights, activations, and research time, and almost no business that rents a model over an API has any of the three. The useful discipline transfers anyway. A neuron description that fits the top examples is a hypothesis until it predicts the rest. An AI-written answer that reads well is the same kind of hypothesis, and the same habit, test before you believe, is what catches it.

RedHub's AI Over-Reliance & Critical-Thinking Check measures that habit from the outside. It seeds a fabricated statistic, plausible-but-false claims, and biased framing into AI text and scores how many a reviewer catches, returning INDEPENDENT, ASSISTED, or RELIANT, with a hard gate: miss the fabricated statistic and the result is capped at ASSISTED regardless of catch rate. No model internals, no AI grading the reviewer, the same marks always produce the same verdict. It is built for individual development and not for employment decisions. What it tests is the skill Gorton applied to a neuron and every reader needs to apply to a paragraph: did the story survive a check, or did it only fit the examples.

Where that skill meets the question of safety, and why legible parts do not add up to a safe system, is the subject of the next post: does interpretability make AI safe?

More in this guide

• AI Interpretability: The Black Box May Be Opening
• Neural Networks Are Already Clean: What the Essay Says
• The Cat and Car Neuron: Solved, or Still Open?
• Does Interpretability Make AI Safe? The Missing Steps

Frequently asked questions

What is mechanistic interpretability?

Mechanistic interpretability is the study of how a neural network produces its behavior in terms of its internal parts, such as neurons, features, and the circuits that connect them, and of what happens when those parts are changed. It differs from asking a model to explain itself, which produces text that may or may not reflect the computation underneath.

What is superposition in neural networks?

In the field's standard sense, superposition is a network representing more features than it has dimensions by spreading features across many neurons, so that a single neuron's activity cannot be read on its own. Goodfire's essay uses a narrower sense, a network packing unrelated things into one neuron because they rarely co-occur, and argues against that. Which sense is meant decides whether the two sides even disagree.

What is a polysemantic neuron?

A neuron that appears to respond to several unrelated things, such as cat faces and car fronts. The appearance can come from a true mixture, from weak tools, from too little context, or from a shared pattern nobody has named yet. Deciding which is the work of interpretability, and the famous cat and car case is still open.

What is a sparse autoencoder?

A second model trained to rewrite a layer's activity as a larger set of rarely active features. It is the main tool for untangling neurons that are believed to be mixtures. Whether it finds structure the neurons lack, or mostly near-copies of single neurons, is one of the points on which Goodfire's essay and its critics report different results.

What is an activation probe?

A small detector trained on a model's internal activations to flag a concept, such as the reward-hacking signal in Goodfire's 2026 study. Probes are cheap to run and can fire before a model acts, but they require access to the model's activations, which a hosted API does not give you.

What is steering?

Changing a model's behavior by amplifying or suppressing an internal direction instead of editing the prompt, as when Anthropic injected "green" into a poem or Goodfire amplified a reward-hacking signal. It is strong evidence that a description is causal. It is also a dual-use capability, since the same handle can push behavior toward or away from what you want.