The AI Inspectability Checklist: 10 Vendor Questions

"Explainable" is on every AI vendor's slide now. Sometimes it means the product shows you its sources. Sometimes it means the model writes a paragraph about why it answered the way it did. Sometimes it means nothing that can be checked. The word is doing a lot of work, and the buyer is usually the one paying for the ambiguity.

This checklist is the practical close to our guide to AI oversight without reviewing everything. The four posts before it covered reward hacking, explanation faithfulness, and who can inspect a model at all. The ten AI vendor explainability questions below are what those posts turn into when you are sitting across from a vendor, or across from your own engineering lead, and need a yes or no.

Two framing notes before the list. First, every question works on your own internal systems as well as on a vendor's. Swap "you" for "we" and ask it in the room. Second, and this is the label the whole post carries: this is a buyer and operator checklist, not an interpretability certification. No one can certify that a model is understood. The research in this series is the reason to be hopeful about understanding, and also the reason to be careful about anyone selling it as finished.

The ten questions

Each question has three parts: why it matters, what counts as evidence, and what is reassurance wearing a tie.

1. What can you see inside the model you run?

Why it matters: inside-the-model inspection, probes, circuits, activations, needs the weights. A vendor built on a hosted API has the model's outputs and nothing else, and so do you. Evidence: a plain answer naming the level. "We call a hosted API. We see outputs, our prompts, and our logs." Or: "We host open weights and can read activations." Reassurance: "Our platform is fully transparent." Ask which level that means. Background: open models, closed APIs, and the right to inspect AI.

2. When your system explains a decision, what is it describing?

Why it matters: a model's stated reasoning can be faithful to its computation or produced after the fact, and the text reads the same either way. Evidence: "The explanation is the retrieved sources and the rule that fired," which is checkable, or "the model's own stated reasoning, which we treat as a claim." Reassurance: "The model explains its reasoning." That is a description of a feature, not of what the feature is evidence of. Background: why an AI's explanation isn't proof of its reasoning.

3. Has the explanation ever been tested against behavior?

Why it matters: a description becomes evidence when it predicts what the system does under a change. Evidence: "We vary the inputs the explanation names and confirm the output moves with them, and here is the test set." Reassurance: "Users find the explanations helpful." Helpful and faithful are different properties.

4. What is your first-line check, and does it run on every output?

Why it matters: oversight that scales starts with a cheap signal that sees everything. Evidence: a named check with a cost per run. A schema check. A policy-word filter. A citation-present check. A probe, if they have the access. Reassurance: "We monitor quality." Monitoring that cannot be named per output is sampling, and sampling has a miss rate nobody has measured.

5. What escalates, to whom, and how fast?

Why it matters: a first-line flag with no owner is a log entry. Evidence: "Flagged items go to a second-model grader within seconds and to a named reviewer within an hour, and here is last month's count." Reassurance: "Issues are escalated appropriately." Appropriately is not a time or a person.

6. Which actions cannot be undone, and who approves them?

Why it matters: model quality does not change the list of irreversible actions. A person must own those regardless of score. Evidence: the list itself, with the approver for each. Reassurance: "The model is highly accurate on those." Accuracy is the wrong axis for an action you cannot reverse. The free Decision Fit Check applies the irreversibility gate to one task in your browser.

7. Can the system reach the thing that grades it?

Why it matters: an agent that can edit its tests, rewrite its acceptance criteria, or close its own ticket has been handed the answer key. That is the opportunity for reward hacking, and no monitor makes a measure trustworthy while the agent can write to it. Evidence: "The grader lives in a system the agent has no write access to, and here is the permission map." Reassurance: "The agent is instructed not to modify tests." Background: reward hacking: when AI passes the test, not the task.

8. What record exists after a decision?

Why it matters: an explanation that cannot be checked later is not one anyone can act on. Evidence: four things, joined: the output, the inputs it saw, the rule in force that day, and who approved it. Reassurance: "Everything is logged." Logs alone do not join those four. Background: what an AI audit trail needs.

9. What changes without telling us?

Why it matters: the model behind a hosted product can be retired, repointed, or retrained, and every check you tuned was tuned on the old one. Evidence: the model name and version, pinned, with a notice period in the contract and the right to re-run your evaluation after any change. Reassurance: "We always use the latest model." That is the problem stated as a feature.

10. Who can stop it, and have they practiced?

Why it matters: a stop that has never been pulled is a hope. Halting mid-task has consequences of its own. Evidence: a named person with the authority, the mechanism, and the date of the last drill. Reassurance: "There is a kill switch." Background: AI agent kill switch and AI agent permissions.

How to use it

1. Send it before the demo. A vendor who sees the questions in advance and arrives with the permission map, the pinned version, and the escalation count is telling you something. So is one who arrives with slides.
2. Write down the answers verbatim. Not your summary. The vendor's words. Reassurance answers are easier to spot in the vendor's own phrasing than in your charitable paraphrase.
3. Get the evidence answers into the contract. Pinned version, notice period, exportable logs, evaluation access, data handling. The checklist is where the terms come from. The contract is where they live.
4. Ask it again at renewal. Answers drift. The person who gave the first set has left. The model has changed twice. The renewal conversation is the second time the list earns its keep.

The one-page PDF has the ten questions with a blank answer column for each, built to be sent ahead of a call and filled in during it.

What this checklist is not

It is not an interpretability certification, and the reason deserves a paragraph, not a disclaimer. The research this series covers shows that some of a model's internal representations can be described, some pathways traced, and some behaviors changed by targeted intervention. Anthropic says its own tracing method "only captures a fraction of the total computation." Goodfire's probe results are for specific behaviors on specific models. The field is still arguing about how far the methods reach, in public and with experiments, which is how a healthy field behaves. A checklist that let a vendor tick "inspectable" would be claiming a finish line the researchers themselves have not claimed. Ten questions that produce a record of what was shown and what was not is the honest version, and it is the version that holds up when someone asks later why you bought.

The outside view at renewal time

The checklist decides whether to trust a system. A different question comes due every twelve months: whether to keep paying for it. The AI Vendor Reliability & Spend-Justification Scorecard grades each AI or SaaS vendor on reliability and whether the spend is justified, from your own status pages, incident logs, invoices, and usage, and returns RENEW, RENEGOTIATE, or DO NOT RENEW. Its reliability-floor gate refuses to justify renewing a business-critical vendor you cannot rely on, however cheap or well-adopted. The gate only holds, never flatters: it can lower a verdict and never lifts a weak one.

For the pre-signature version of the same discipline, the AI Vendor Claim & Contract Scrutiny Kit scores each claim on whether it is proven and whether it is in the contract, and forces WALK AWAY on an ROI or autonomy promise that is neither. Both grade your marks and nothing else. Neither reads the vendor's model, and neither claims to.

If you own a RedHub instrument, the MCP connector lets Claude list and explain it in the chat, and run it when it ships a runnable engine, returning the verdict, whether the gate fired, the fix-first when there is one, and the instrument's own disclaimer. It reads only what you own and cannot buy anything.

Keep going: the oversight design these questions come from is in AI oversight without reviewing everything. The research behind the caution in question two is in why an AI's explanation isn't proof of its reasoning, and the access problem behind question one is in open models, closed APIs, and the right to inspect AI. For what is new inside the models, and why the hope is real, start at AI's black box may be opening.