Unstructured to Structured Data: A Working Pipeline

Most business data starts life unstructured. Invoices arrive as PDFs. Forms come in as scans. Contracts, statements, delivery notes — all documents a human can read and a database can't. Turning unstructured to structured data is the unglamorous conversion underneath almost every ops automation you'd actually want to build.

Language models made the middle of that conversion easy. Point one at an invoice and it will hand you tidy JSON. What the model can't do is tell you whether that JSON is safe to load — and a pipeline that skips that question isn't a pipeline. It's a hope. Here's the full five-step design, in plain English.

Step 1: Define the schema — decide what "structured" means

A schema is a short list: field name, field type, required or optional. It's the contract the rest of the pipeline enforces. For an invoice: invoice number (text, required), date (date, required), vendor (text, required), total (number, required), currency and line items optional.

Three rules make schemas work in practice:

• Start small. Five or six fields you'll actually use beat twenty you might. Every field you add is a field that can flag.
• Be strict about types. "Date" means a value that parses as a date — not whatever text sat near the word "Date" on the page. Types are what let validation do real work.
• Require sparingly. Mark a field required only if its absence should stop the document. Required-everything turns your review queue into a wall.

Step 2: Extract — let the model fill the schema

The document text and the schema go to the model; the model returns its best value for each field. This is the commodity step — any current model can do it. The one non-negotiable: the extraction should return a confidence rating per field (high, medium, low), not just values. Without per-field confidence, every downstream decision is blind.

Step 3: Validate — check the machine's work mechanically

Validation is deterministic — no AI involved, which is exactly why you can trust it:

• Every required field is present.
• Every date parses as a date.
• Every number is actually numeric.

These checks are cheap, fast, and incorruptible. A model can be confidently wrong; a date parser cannot be talked into accepting "N/A" as a date. This is the deterministic backstop that blind-trust pipelines skip. Field-level validation goes much deeper than three checks — cross-field rules, ranges, format patterns — and that deeper layer is the lane of the Document Field Validator ($79).

Step 4: Flag — make uncertainty visible

Anything low-confidence, missing, or invalid gets marked REVIEW. Not corrected silently. Not dropped. Marked, so a human sees exactly which fields need eyes — and only those. The flag is the pipeline's honesty made operational: instead of one confident-looking blob of output, you get a per-field verdict you can route on. How much gets flagged, and how to track whether the flags are working, is covered in document extraction accuracy: what to expect and verify.

Step 5: Gate — let the exit code decide

The last step is the one that makes this a pipeline instead of a script: the extractor's result controls what happens next mechanically. The convention is simple — exit code zero when every field validated and came back confident, non-zero when anything needs review. Whatever runs the extractor branches on that code:

This is how the runnable extractor in the AI Document Extraction Kit is built: extract.py exits non-zero whenever any field needs review, so it drops into a scheduled job, a folder watcher, or an automation platform without any glue logic. The gate means no human forgets to check — because the pipeline physically can't proceed on a flagged document.

Before and after the pipeline

Extraction converts one document type into rows. A real document operation has stages on either side:

• Before: mixed documents need sorting to the right schema first — the Document Classify & Route Kit ($69) handles classify-and-route ahead of extraction.
• After: structured rows are only the start — the AI Data Analysis Starter Kit ($79) turns clean rows into defensible analysis.
• Across the whole chain: if you're not sure which stage leaks — intake, classification, extraction, validation, or load — the Document Processing Pipeline Diagnostic ($79) scores the full pipeline and names the weakest link.

If your first document type is invoices — and for most teams it should be — the AP-specific walkthrough is in invoice data extraction: automate AP without bad data.