How Quickly Your Evidence Stops Being True
RedHub AI Editorialupdated September 2, 20266 min read
Jump to a section8
Evidence goes stale when the thing it describes changes and the record does not. Nothing is deleted and nothing is wrong on the day it is written — the system moves underneath it. The practical fix is to give every artifact a review interval matched to how fast its subject turns over, not a uniform annual review that is too slow for some documents and pointless busywork for others.
TL;DR: A document with no date on it cannot be assessed for freshness at all, which is the most common version of this problem. Rank your artifacts by the turnover rate of what they describe, then set intervals to match. This is the "change" axis of the AI audit trail guide.
Stale Is Not the Same as Missing
A missing record announces itself. You go looking, it is not there, everybody understands the situation immediately.
A stale record is worse, because it answers. It gives a confident, specific, wrong account of how the system works, and it does so in a document that looks maintained. Somebody relies on it. Somebody quotes it to a customer. The error is only discovered when it collides with reality, usually in front of the person you least wanted to discover it with.
The uncomfortable version: a stale evidence pack is more dangerous than no evidence pack, because no pack produces caution and a wrong one produces confidence.
What Drifts
Not everything ages at the same rate, and treating it as if it does is why annual review cycles fail. Four things move fast enough to matter.
Prompts and thresholds
The fastest-moving thing in most AI systems, and the least likely to be versioned. A prompt can change weekly, each change is a small improvement, and no single change feels like an event worth recording. After six months the document describing system behavior is describing a system that no longer exists.
Model versions
Sometimes you change these. Sometimes a vendor does, on their schedule, and the first you hear of it is a deprecation notice or a change in output. A record that names a model family without a version pins nothing.
Data sources and retrieval corpora
A re-index, a new source, a changed filter. The system now sees a different world than the one your documentation describes, and this is a hard one to notice, because nothing about the application changed.
People
Approvers leave. Owners move teams. A named accountable person who left eighteen months ago is a record that reads as complete and resolves to nobody, which is the failure that turns into the bus-factor problem from the other direction.
Match the Interval to the Turnover
The useful move is to stop asking "when did we last review this" and start asking "how fast does what this describes change." Then set the interval from the answer.
- Things that change weekly — prompts, thresholds, routing rules. These need versioning rather than review. A review cycle can never keep up with a weekly change, so the record has to be produced by the change itself.
- Things that change a few times a year — model versions, data sources, vendor terms. A quarterly check is proportionate.
- Things that change rarely but consequentially — approvals, ownership, the policy itself. Annual is fine, provided the review is real rather than a re-dated signature.
The failure mode to avoid is the uniform annual review, which is simultaneously far too slow for the first group and a waste of everyone's time on the third.
Undated Is the Common Case
Before any of that helps, most artifacts need a date at all. A document with no date cannot be assessed for freshness — it is neither fresh nor stale, it is unassessable, and it will be treated as current by whoever finds it.
Three dates are worth carrying, and they are different things:
- Written. When this was first produced.
- Last reviewed. When somebody read it against the system and confirmed it still holds. Not when somebody edited a typo.
- Review due. The date the interval above produces.
The second one is where honesty is required. A review date that records a file being touched instead of checked is worse than no date, because it manufactures confidence that nobody earned.
A quick audit: open the five documents you would hand a customer who asked how your AI systems work. Count how many carry a real last-reviewed date. In most teams the answer is zero or one, and that is the finding.
The Condition Worth Checking Across the Whole Set
There is a pattern that individual-document review cannot see. Every artifact scores reasonably, each one is organized and plausible, and not one has been reviewed since it was written. Judged one at a time they look fine. Judged as a set, nobody has looked at any of it.
That is a property of the practice rather than of any single record, which is why it needs a question asked about the whole group: does anything here carry a date somebody has honored? If the answer is no across every artifact, the state of any individual one stops being the interesting fact.
Grading It
The Audit Evidence Pack Assay ($119) grades whether an account of a decision can be assembled at all — 24 questions on the joins between the record and the context, policy and authorization behind it, returning ANSWERABLE, NEEDS A WITNESS or NO RECORD. It grades what you describe; it is not a scanner, and every input is an answer you supply about your own wiring.
Freshness specifically is one of the eight domains in the free AI Compliance Assurance assessment, and it carries the set-level condition described above: when nothing in that domain records a review that actually happened, the top verdict is held at DRIFTING however well the rest scores, because a good number there would be describing a snapshot instead of a standing position. It measures durability, not adequacy, and it is a self-assessment rather than an audit.
Pairs well with
Decide how long each record type is kept, deliberately, with the Retention Purge Scheduler ($59); if your disclosures to customers are among the artifacts drifting, the AI Disclosure & Synthetic-Content Labeling Readiness Kit ($79) covers that surface.
This is general operational guidance, not legal advice, and nothing here states what any law requires of your records or review cycles. Ask a qualified lawyer about your obligations.
More in this guide
What does it mean for evidence to go stale?
The system it describes changed and the record did not. Nothing was deleted and nothing was wrong when written, which is what makes it hard to notice: the document still looks maintained and now gives a confident wrong account.
Why is a stale record worse than a missing one?
A missing record produces caution, because everybody can see the gap. A stale one produces confidence, because it answers. People rely on it and quote it, and the error surfaces when it collides with reality in front of somebody who matters.
How often should we review documentation?
Set the interval from how fast the subject turns over, not from a calendar. Weekly-changing things like prompts need versioning, not review, because no review cycle can keep pace. Quarterly suits model versions and data sources; annual suits approvals and ownership.
What is the most common problem here?
No date at all. An undated document cannot be assessed for freshness — it is neither fresh nor stale — and whoever finds it will assume it is current.
Does a last-reviewed date mean the file was edited?
It should not. A review date records that somebody read the document against the system and confirmed it still holds. A date that only records a file being touched manufactures confidence nobody earned, which is worse than leaving it undated.
The gate this post refers to, drawn from the tool’s own logic. See the tool.