Open Models, Closed APIs, and the Right to Inspect AI

Look at where the research in this series was done. Goodfire's reward-hacking probes ran on Kimi K3, GLM 5.2, and Qwen 3.8 Max, which the lab describes as "three of the most capable open-source models." Nick Cammarata's neuron audits in the Already Clean essay (opens in a new tab) ran on Qwen3-32B. Transluce described every neuron in Llama-3.1-8B-Instruct. Anthropic traced circuits in Claude 3.5 Haiku, its own model. Every one of those teams could read the activations. That is not a coincidence. It is the entry fee.

If your AI is a hosted API, or a vendor's product wrapped around one, you did not pay that fee and cannot. You get text in and text out. The ability to inspect AI models from the inside, which is what makes the research in this series exciting, is not distributed evenly, and the question of who has it is a business question before it is a technical one. This is the third cluster in our guide to AI oversight without reviewing everything.

Three levels of access

There are three places you can stand relative to a model, and each one decides what you can check.

The middle row is where the open-model results came from, and it is available to any business willing to host a model. Goodfire's footnote on infrastructure is worth noting: the lab writes that frontier labs, naming OpenAI, Anthropic, and Google DeepMind, "actually already use activation probes for monitoring." So the top row is monitoring itself from the inside. The question for everyone else is whether the middle row is worth the move.

What open weights change

Hosting an open-weight model gives you three things a hosted API never will.

You can read the inside. Activations are available on every forward pass. A probe like the ones in Goodfire's research is, in the lab's description, a small detector that reads them. You could train one. Cammarata's essay reports that "In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job," with caveats about thresholds and depth that the essay spells out. Whatever the final number turns out to be, the audit was possible because the weights were open.

Nothing changes under you. A pinned open-weight model is the same model next year. A hosted API's model can be retired, repointed, or retrained, and the vendor's notice period is whatever the contract says. The earlier post MCP model portability covers what happens to your tooling when a hosted model retires.

You can test the claims yourself. Independent replication is how the field is checking the Already Clean essay right now, and that replication was possible for the vision models because the models were available. A claim you can test is worth more than a claim you have to take on faith, and that holds for a vendor's claims about its own model as much as for a research essay's. The research-literacy version of that argument is in what Goodfire's Already Clean essay says.

What open weights do not change

Here is the complication, and it is the reason most businesses will stay at the first level for a while. Access is not understanding.

Reading activations is the start of a research program, not the end of one. Goodfire's probes are simple by the lab's own account, which says it "used only the simplest possible techniques" (opens in a new tab), but they still needed the model's internals, examples of the behavior to train on, and a benchmark to validate against. Anthropic reports that understanding a traced circuit "takes a few hours of human effort" even on short prompts. Transluce's neuron descriptions describe what a component responds to and, in the lab's own words, do "not describe how model decisions are made." The middle row gives you the right to look. It does not give you the staff, and the staff is the expensive part.

There is also a security cost that cuts the other way. An open-weight model you host is a model you have to defend yourself. The same access that lets you probe it lets an attacker who reaches your infrastructure steer it. Whether your model can be made to misbehave at the prompt, output, and data-handling layer is the subject of llmsecurityaudit.com (opens in a new tab), a free self-assessment you answer yourself, with no sign-up.

So the honest framing is this. Open weights move the ceiling on what you could inspect. They do not move the floor on what you will inspect unless you fund the work. For most businesses, the practical gain is the second item above, version stability, and the third, the option to verify a claim. Those are real. They are not the same as an inside view.

The vendor in the middle

Most businesses do not buy models. They buy products built on models, and the vendor selling the product is often standing at the same level you are: output only, through someone else's API. That has a consequence worth stating plainly. A vendor cannot sell you inspection it does not have. When a product built on a hosted API says it is "explainable" or "transparent," what it can mean is that it shows you its prompts, its retrieval sources, its logs, and the model's own stated reasoning. It cannot mean activations, because the vendor never sees them either.

That is not a reason to distrust the vendor. Prompts, sources, and logs are the outside view, and the outside view is what this whole series says a business can run. It is a reason to ask what, specifically, the vendor can see, and to write down the answer. Seven signs that one model owns your workflow, and the contract terms that follow, are in AI vendor lock-in.

What "the right to inspect" means in a contract

You will not get activations from a vendor. You can get the outside-view equivalents in writing, and the ones that matter are short. This is procurement hygiene, not legal advice, and your own counsel should read anything you sign.

1. Which model, pinned. The name and version behind the product, and notice before it changes. A product that cannot tell you which model it runs cannot tell you why its behavior moved.
2. Logs you can export. Inputs, outputs, and the retrieval sources or tool calls for each, in a format you can take with you. Logs the vendor holds and you cannot export are the vendor's audit trail, not yours.
3. Evaluation access. The right to run your own test set against the product and to re-run it after any model change. Without this, "we evaluated it" means they did, once.
4. Incident disclosure. When the model behind the product fails in a way that affects customers, who tells you, how fast, and what you receive.
5. Data handling, in writing. Whether your data trains anything, and how you get it back and deleted when you leave. These two have no acceptable verbal answer.

Three RedHub instruments grade exactly these conversations, from your own reading of the pitch and the paperwork. The AI Vendor Claim & Contract Scrutiny Kit scores each vendor claim on two axes, whether it is proven and whether it is in the contract, and the per-claim verdict follows the weaker of the two, so a claim that is proven but unwritten cannot read SUBSTANTIATED. Its high-stakes gate forces WALK AWAY when an ROI or autonomy promise is both unproven and not in the contract, no matter how clean the rest of the pitch. The AI Vendor Security Posture Scrutiny Kit scores six security controls and returns SOLID, PRESS FOR PROOF, or RED FLAG, with a disqualifier gate that forces RED FLAG when a vendor trains on your data with no opt-out or offers no deletion and exit path. And if you are the one publishing a model or system card, the Model & System Card Builder grades seven disclosure sections DISCLOSED, THIN, or MISSING and returns PUBLISHABLE, DRAFT, or NOT PUBLISHABLE, with a gate that forces NOT PUBLISHABLE on a performance or fairness claim the card cannot back.

For what the AI already in use across your organization exposes you to, what the vendors hold and what leaves through it, genaisecurityassessment.com (opens in a new tab) is a free self-assessment you answer yourself.

The honest boundary

Nothing in this post gets you an inside view of a model you reach through an API, and no instrument RedHub sells reads a model's internals. What the outside view gets you is a clear statement of what your vendor can and cannot see, a contract that reflects it, logs you own, and the right to test. That is the inspection available at the first level, and it is a lot more than most AI contracts currently secure. One scenario, not a forecast, is that the research in this series makes the inside view better and, over time, more widely available. Even then, it is not a reason to sign for one today.

Keep going: the tiered oversight design that runs at every access level is in AI oversight without reviewing everything. The previous cluster, why an AI's explanation isn't proof of its reasoning, covers the test ladder that needs this access. The next turns all of it into ten questions: the AI inspectability checklist. For the security posture that should sit around any model you do host, read zero trust for AI.