Document Extraction Accuracy: What to Expect and Verify

Ask a vendor about document extraction accuracy and you'll usually get a big clean number. Ask what documents that number was measured on, and things get quieter. Benchmark accuracy is measured on benchmark documents — and your documents are not benchmark documents. They're scanned at an angle, stamped, coffee-stained, formatted by a vendor whose invoice template was last updated years ago.

This post is the honest version: what actually drives extraction accuracy, why a single number misleads, and the two metrics that tell you the truth about your own pipeline.

What actually drives accuracy

Notice the pattern: the failure zone clusters around totals, dates, and handwriting — the high-stakes fields. A model can be excellent at reading vendor names and still misread the one number you were automating this to get right. That's why an average accuracy figure hides more than it reveals: it blends the easy fields with the dangerous ones.

The two numbers that actually matter

1. Flag rate

Flag rate is the share of documents where at least one field came back low-confidence, missing, or invalid — and got marked REVIEW instead of accepted. It's the system telling you, honestly, how much of your document stream it can't vouch for. A healthy flag rate is neither zero nor huge: near-zero usually means validation isn't really running, and a very high rate means your inputs are messy or your schema is over-strict.

2. Sampled accuracy

Flags catch what the model knows it's unsure of. Sampled accuracy catches what it doesn't. Each week, pull a share of the documents that auto-accepted, check them against the source, and record the result. That running number is your real accuracy — measured on your documents, not a vendor's benchmark. If it drifts down, you find out in a week, not at year-end reconciliation.

1. Log every processed document with its outcome: auto-accepted or flagged.
2. Compute flag rate weekly — flagged ÷ total. Watch the trend, not just the level.
3. Sample the auto-accepted pile — pick a fixed share, verify each field against the source document.
4. Record sampled accuracy and investigate any drop: new vendor template? Worse scans? Schema drift?

This is exactly the structure of the QA Tracker that ships in the AI Document Extraction Kit: a documents log, field-level QA, and a dashboard for flag rate and sampled accuracy. The point isn't the spreadsheet — it's the discipline. Accuracy is a number you keep measuring, not a claim you buy once.

What review actually costs you

The trade a validate-and-flag workflow offers: instead of a person touching every document, a person touches only the flagged ones (plus a small audit sample). Try your own numbers — this is arithmetic on your inputs, not a measured claim:

The gap between those two numbers is what you're buying with automation — and the flag rate is what keeps the smaller number honest. Automation with no flags would show an even smaller number, but it would be borrowing against errors you haven't found yet.

Red flags when a vendor talks accuracy

• One number, no context. Accuracy without "on what documents, on which fields" is marketing, not measurement.
• No confidence output. If the tool can't tell you which fields it's unsure of, you can't route review — you can only trust or re-check everything.
• No failure mode. Ask what happens when a required field is missing. "It fills in its best guess" is the wrong answer.
• Accuracy guarantees. A guarantee on model output over unknown future documents isn't a guarantee — it's a wish with a logo.

Where does the human fit once the metrics are in place? That's the review-design question — covered in human-in-the-loop document processing, done right. And if you're starting from invoices specifically, the field-by-field walkthrough is in invoice data extraction: automate AP without bad data.