AI in Law, Health and Finance: What Proof Looks Like

A lawyer, a clinician and a financial adviser all got the same gift this year: a tool that reads faster than they do and explains itself in clean, confident paragraphs. The explanation is the problem. “Explainable AI” is sometimes presented as the model telling a user why it answered as it did. That tells you what the model wrote about its answer. It does not tell you the answer is right. Proof is something else: a record a second person can check without trusting the first. This is the professional chapter of our series on what would prove real AI safety progress, and it is the one chapter where the stakes land on a named person.

This post is general information, not legal, medical or financial advice. Qualified professionals and counsel make the determination in any real matter. Where a law, regulator or professional standard is named below, the description is a summary of a published source as of October 2026, and you should confirm its current text before relying on it.

Why the model's explanation is not evidence

Start with what the people who can see inside a model say about it. Anthropic's interpretability team traced the internal steps behind some of Claude's answers and published the limits plainly: "our method only captures a fraction of the total computation performed by Claude" (opens in a new tab). In the same work they found cases where the written reasoning matched the internal steps, and cases where, given a hint, the model worked backwards from the hinted answer and wrote reasoning to fit. Anthropic notes that the made-up reasoning "can be very convincing" (opens in a new tab).

That is the whole issue in one finding. A model's explanation of its answer is not sufficient evidence of how the answer was produced. Sometimes it is faithful. Sometimes it is a story. From the outside, you cannot tell which, and an ordinary hosted-model API does not give the customer access to the model’s internal computation sufficient to verify that a written explanation faithfully describes how the answer was produced. Our post on why an AI's explanation is not proof of its reasoning covers the research in depth, and the series pillar on interpretability covers what is improving on the inside. Both are hopeful. Neither is available to a law firm, a clinic or an advisory practice using a vendor's model this year.

So the question for a professional is not "did the model explain itself." It is "what can I show." That has an answer today.

What proof looks like: four questions

A record that holds up answers four questions, and the four have to be joined to the same piece of work. This is the same shape as the four joins in what an AI audit trail needs beyond logs, applied to one deliverable instead of one system.

1. What did it see. Which documents, which versions, which fields, on what date. A common failure in professional work is a scope error: the tool answered correctly from the wrong or incomplete input. Without this record, that failure is invisible.
2. What did it claim, and where does each claim point. For the record recommended here, each material claim on which the professional’s decision relies should point to a source the reviewer can open: a clause, a chart field, a client note. A claim with no pointer is an assertion, not a finding.
3. Who checked it, against what. A named person compared the output to the source. Not "reviewed." Compared, with the specific fields or clauses named. A signature alone does not show what substantive review occurred, and it may create a record showing that the named person approved a draft that still contained the error.
4. What survives. The record has to outlive the tool, the prompt, the model version and the staff change. Retention periods and permissible contents vary; preserve records under the organization’s applicable legal, privacy and records-management requirements. Logs are not an audit trail for the same reason a stack of outcomes is not an explanation.

None of that requires a single neuron. All of it is checkable by a person who was not in the room.

Three hypothetical scenarios

Each scenario below is invented to show the shape of the failure and the shape of the proof. None describes a real matter, patient or client, and none offers advice on how to handle one.

Law: the appendix that changed the answer

Hypothetical scenario. A firm's AI assistant summarizes a 60-page services agreement for a partner who has twenty minutes. The summary says the liability cap is twelve months of fees and applies to the whole agreement. It cites section 14.2, and section 14.2 says exactly that. Appendix C, incorporated by reference on page 58, carves data-breach costs out of the cap. The assistant never read Appendix C, because the file it was given was the main body exported without its schedules. The summary is fluent, specific, correctly cited and wrong on the one point the client cares about.

Notice what catches it. Not a better model. The record of what the tool saw: a file list that shows the main body and no schedules. A reviewer who checks "which files fed this" before checking any clause finds the gap in one minute. A reviewer who checks the clause finds nothing wrong, because the clause is right.

On the professional side, the American Bar Association Standing Committee on Ethics and Professional Responsibility’s Formal Opinion 512 (July 2024) discusses lawyers’ use of generative AI under the ABA Model Rules, including competence, confidentiality, communication with clients, supervision of lawyers and nonlawyers, meritorious claims and candor to tribunals, and fees.

Formal Opinion 512 is an ethics opinion from the ABA Standing Committee interpreting the ABA Model Rules. It is not itself binding law or a court rule. A lawyer’s enforceable duties depend on the professional-conduct rules and other authorities applicable in the lawyer’s jurisdiction, and state and local authorities may issue guidance that differs.

Finance: the constraint the assistant never saw

Hypothetical scenario. An adviser's AI assistant drafts a rebalancing proposal from a client's holdings and the firm's model portfolio. The draft recommends trimming a concentrated position in the client's former employer. In a March call, the client said that stock cannot be sold before a lock-up ends in November, and the adviser wrote it in the CRM. The CRM note was not in the file the assistant was given. The draft follows the model portfolio but conflicts with a client constraint recorded in the CRM and omitted from the assistant’s inputs. Nothing here is investment advice, and the numbers do not matter. The missing input does.

This hypothetical does not determine whether Regulation Best Interest, an investment adviser’s fiduciary duty, a suitability standard, a contractual restriction or another obligation applies. That depends on the professional, firm, account, recommendation and applicable law.

The proof is a step with a name: the draft is checked against the client's stated constraints, listed by name, before it leaves the adviser's desk, and the record shows that check happened and who did it. The explanation the assistant wrote for its recommendation never mentions the lock-up, because the assistant never knew about it. An explanation can only cover what the model saw.

For FINRA member firms, FINRA, the self-regulatory organization for broker-dealers, has addressed the use of generative AI. Regulatory Notice 24-09 (opens in a new tab) (June 27, 2024) states: “In addition, FINRA reminds its member firms that FINRA’s rules–which are intended to be technology neutral–and the securities laws more generally, continue to apply when member firms use Gen AI or similar technologies in the course of their businesses, just as they apply when member firms use any other technology or tool.” The notice also states: “This Notice does not create new legal or regulatory requirements or new interpretations of existing requirements, nor does it relieve member firms of any existing obligations under federal securities laws and regulations.” The notice is addressed to FINRA member firms. Whether it, or a different regulatory framework, reaches a particular firm—and what that framework currently requires—is a question for the firm’s compliance professionals and counsel.

Health: the summary that dropped a line

Hypothetical scenario. A clinic uses an AI tool to draft an after-visit summary from the clinician's dictation and the intake form. The intake form lists a penicillin allergy. The dictation never mentions it, because the visit was about a knee. The draft summary omits the allergy. No diagnosis is made here and no treatment is chosen. The only question is whether the record is complete, and the draft reads as complete because it is smooth.

The proof is boring on purpose. Before signing, the clinician compares the draft to its sources using the clinic’s documented list of fields designated for verification, including allergy information where the clinic’s workflow requires it, and the signed note records which source documents fed the draft. A separate question comes before any of this: whether patient information may be provided to that vendor is a privacy, security and contracting matter for the clinic’s compliance office and counsel, addressed before the workflow is used.

What the sources have in common, and where this breaks

Read the two sources named above together and one pattern shows. Each is addressed to the professional and the firm, under rules that already governed the work: for the ABA opinion, duties arising under the Model Rules discussed in the opinion; for the FINRA notice, the firm's existing obligations, supervision among them. Neither is written as an instruction to explain what happens inside the model.

Some regimes separately require an affected person to receive reasons or information about a decision. For example, ECOA and Regulation B require creditors to provide specific principal reasons for certain adverse actions, and Article 86 of the EU AI Act provides a right to a clear and meaningful explanation of the role of certain high-risk AI systems and the main elements of certain decisions. GDPR transparency and automated-decision provisions may impose related but distinct information and safeguard requirements. Those duties are not interchangeable, and they are not necessarily demands to disclose a model’s internal chain of thought. Which rule applies, what explanation is required and when it must be provided are questions for qualified counsel.

The record this post describes is our recommendation for showing that work, not a requirement either source states. In the EU, the AI Act is binding law with provisions that apply on a phased schedule. Regulation (EU) 2026/1744 amended the Act and changed parts of that schedule. Our dated EU AI Act correction post summarizes those changes, but organizations should confirm the current consolidated text, their role and the system’s classification with qualified counsel before relying on any date or obligation.

Two complications, and this post does not resolve either.

First, the review step is where the process can quietly fail. A reviewer under time pressure who signs a fluent draft may create a record showing that the named person approved a draft that still contained the error. A signature alone does not show what substantive review occurred. The record should identify what was compared, against which source and by whom. A firm that cannot afford that comparison on every output has to decide which outputs get it. That is a resourcing decision nobody can make for you, and the oversight pillar is about making it on purpose.

Second, the outside view proves less than people want it to. It cannot show the model reasoned well. It can show the output matched the source this time, for this input, checked by this person. That is a narrower claim. It is also the claim that holds up when someone asks a hard question later, and it does not depend on any research result arriving on schedule.

The outside view you can run this week

Everything below grades your own marks about your own work and hands back a verdict and, where something is short, the one thing to fix first. None of it reads a model. None of it certifies anything, and none of it makes a firm compliant with anything.

• Should a model make this call at all. The free Decision Fit Check places one task on a five-level ladder, from deterministic code up to a person. Two gates can push a task toward a human and never away from one: if two competent reviewers could not agree on the right answer, or if a wrong answer cannot be undone and nobody sees it first. The tool is designed to perform the assessment in the browser. Verify the current tool’s data handling and network behavior before entering sensitive information.
• Does every claim in this deliverable point somewhere real. The AI Deliverable Claim & Citation Defensibility Gate grades a finished AI-assisted deliverable claim by claim and returns DEFENSIBLE, VERIFY FIRST or DO NOT SHIP. The product’s top label is DEFENSIBLE. In this instrument, that label means the entered citation-evidence marks satisfy the product’s stated scoring and gate rules. It does not mean the document is legally defensible, accurate in every respect or approved by counsel. In its worked sample a board memo scores 89 out of 100 and still reads DO NOT SHIP, because two material claims rest on citations a reviewer cannot open or that show signs of fabrication. You do the reviewing and enter the marks. The engine applies the gate. Its own page calls it a review aid for your own deliverables, not legal advice.
• If the AI decides something about a person, were they told. The Automated-Decision Transparency & Disclosure Scorecard grades a decision system on six transparency and recourse practices and returns DISCLOSED, GAPS, UNDISCLOSED or MATERIAL TRANSPARENCY GAPS. These are product-defined readiness results, not compliance determinations. Its insurance-tier preset scores 74 and still reads UNDISCLOSED, because the disclosure control is marked zero: nothing shows the person was told an automated system decided. Its page is explicit that it renders no compliance ruling and is a readiness aid, not legal advice.
• Where is the organization short of a named framework. aicompliancegapassessment.com (opens in a new tab) is a free self-assessment you answer yourself, measured against NIST AI RMF, ISO/IEC 42001 or the EU AI Act. Its own page carries the scope line: "NIST AI RMF and ISO/IEC 42001 are voluntary standards; the EU AI Act is law. This assessment measures alignment; it does not determine whether a framework applies to you."

Several of these tools use a gate that can move a score-derived result toward caution but never improve it. In the examples above, the gated condition controls the displayed result despite the numerical score. The deliverable at 89 still does not ship. The decision system at 74 is still undisclosed. That is what proof looks like from the outside: the one thing that is wrong holds the whole thing, however good the average.

Keep going: the tools that let researchers steer a model toward safety also let someone steer it away, which is the subject of the security risk of editing an AI's behavior. And a complete record still leaves one question open, who decides, which is where transparency is not accountability picks up. If you are buying a tool that claims to be explainable, take the ten vendor questions into the meeting.