Why AI Detectors Don't Work (and What to Do Instead)

In July 2023, OpenAI withdrew its own AI text classifier (opens in a new tab), saying it was no longer available "due to its low rate of accuracy." That is the short answer to why AI detectors don't work as proof of who wrote something. They guess from statistical patterns. Those guesses flag human writing and miss edited AI text often enough that a score settles nothing. For a publisher, the better question is whether the draft is worth reading, and that one you can answer from the text.

TL;DR: OpenAI's classifier caught 26% of AI-written text in its own tests and wrongly flagged 9% of human text before OpenAI pulled it. A 2023 Stanford study found seven detectors flagged non-native English essays as AI-written at an average rate of 61.22%, while scoring US eighth-grade essays almost perfectly. One editing prompt cut detection of AI-written essays from 100% to 13%. Treat a detector score as a weak signal at most, and grade quality instead of origin. The Anti-Slop Content System ($79) is built on that stance. The AI slop guide covers what to look for in the text itself.

What OpenAI's own classifier showed

OpenAI launched its classifier on January 31, 2023. The launch post reported results on a "challenge set" of English texts. The classifier labeled 26% of AI-written text "likely AI-written," and it labeled human-written text as AI-written 9% of the time. OpenAI called it "very unreliable on short texts (below 1,000 characters)" and said it "should not be used as a primary decision-making tool."

On July 20, 2023, OpenAI added a note to the same page: the classifier was no longer available due to its low rate of accuracy. The company behind ChatGPT could not make its own detector reliable enough to keep.

Who gets flagged

A 2023 study by Stanford researchers, published in the journal Patterns under the title "GPT detectors are biased against non-native English writers (opens in a new tab)," tested seven widely used detectors. The researchers ran them on 91 TOEFL essays by non-native English writers and 88 essays by US eighth graders. On the eighth-grade essays, the detectors were near-perfect. On the TOEFL essays, they flagged an average of 61.22% as AI-generated. Every one of those essays was written by a person. 89 of the 91 TOEFL essays were flagged by at least one detector.

That near-perfect result on the native essays is the uncomfortable part. A detector that fails at random would be easy to ignore. These detectors failed on a particular group of writers: people writing in a second language, who tend to use a narrower range of words, according to the researchers. The researchers tied it to perplexity, a measure of how predictable the text is to a language model. Plain writing is predictable, so it reads as machine-made to a tool built on that measure. When they had ChatGPT simplify the word choices in the eighth-grade essays, the average misclassification rate rose from 5.19% to 56.65%.

Detector vendors have released newer models since 2023 and publish their own accuracy figures. Those figures come from the vendors' test sets, not your writers. The pattern above is the one to test for before you trust any of them with a decision about a person.

A small error rate at volume

Turnitin added AI detection to its product in April 2023 and, according to Vanderbilt University (opens in a new tab), claimed a 1% false positive rate at launch. Vanderbilt did the arithmetic in an August 2023 notice: it had submitted 75,000 papers to Turnitin in 2022, so about 750 student papers could have been wrongly labeled as partly AI-written. Vanderbilt disabled the detector "for the foreseeable future."

The same math applies to a content team. At 1%, a team that checks 200 human-written drafts a month should expect about two of them flagged as AI every month, and that is using the vendor's own number.

Editing defeats detectors in the other direction

False positives are half the problem. The Stanford team also generated 31 college admission essays with ChatGPT-3.5, which the detectors caught at first. Then they asked the model to "elevate the provided text by employing literary language." Detection fell from 100% to 13%.

So a detector flags plain human writing and misses AI writing after one more prompt. A score that can be wrong in both directions cannot tell you which direction it is wrong in today.

The goal was never to fool a detector

A draft that games a detector into a "human" score but still reads generic has solved nothing. Readers don't run a detector before deciding whether to keep reading. They just stop. Writing to pass a detector also pulls in the wrong direction, toward ornate phrasing that changes a statistic, when the reader wanted a fact, a position and a voice.

The same logic holds well outside publishing. Our post on AI-generated resumes makes the case for checking what a document claims instead of guessing who wrote it.

What to use instead

Ask "is this worth reading?" instead of "did AI write this?" SPARK is built to answer the first question. It rates a draft 0 to 5 on Specific, Point of view, Authentic voice, Real and Kinetic structure, and the weighted total returns Publish, Revise or Rework. None of those marks care how the first draft was produced. To move a team off a detector-first process:

• Stop routing drafts through a detector as the first or only check.
• Score every draft on quality before it publishes. A low rating names the fix. A detector score names nothing.
• If you still run a detector, record the score as one weak signal and never as a pass or fail.
• Never act against a person on a detector score alone.

Pairs well with

The AI Deliverable Claim & Citation Defensibility Gate ($79) checks what a detector never does: whether each claim in a finished deliverable is cited, reachable and supported. It returns DEFENSIBLE, VERIFY FIRST or DO NOT SHIP, and it grades claims, never people. The Brand Voice Engine ($59) gives reviewers a written Voice Spec and a 100-point QA Checker to judge drafts against. Once your content is specific enough to cite, the GEO / AI Visibility Playbook ($149) covers structuring it for AI answer engines, with no guaranteed citations.

More in this guide

• AI Slop: What It Is and How to Stop Writing It
• How to Make AI Content Sound Human
• Signs of AI-Generated Content: Craft Flaws, Not Proof
• How to Keep AI Content Quality High at Scale

Do AI content detectors work?

Not as proof. OpenAI withdrew its own classifier in July 2023 for low accuracy, and a 2023 Stanford study found seven detectors flagged non-native English essays as AI-written at an average rate of 61.22%. A score is a guess, not evidence.

Why did OpenAI shut down its AI classifier?

OpenAI said the classifier was no longer available as of July 20, 2023, due to its low rate of accuracy. In OpenAI's own tests it identified 26% of AI-written text and wrongly flagged 9% of human-written text.

Are non-native English writers flagged more often?

In the 2023 Stanford study, yes. Seven detectors were near-perfect on US eighth-grade essays but flagged an average of 61.22% of TOEFL essays by non-native writers as AI-generated, and 89 of 91 were flagged by at least one detector.

Can editing get AI text past a detector?

It did in the 2023 Stanford study. One prompt asking ChatGPT-3.5 to rewrite its own essays in more literary language cut detection from 100% to 13%. A detector can miss edited AI text as easily as it flags plain human writing.

Should I still run my content through a detector?

You can, but record the result as one weak signal, never as a pass or fail, and never act against a writer on a score alone. A quality scorecard tells you what to fix, and a detector score does not.

Does the Anti-Slop Content System use an AI detector?

No. It uses a heuristic scanner and the SPARK scorecard to flag quality problems for revision. Neither one certifies text as human or AI, and the kit treats any detector score as a signal, not proof.