LLM Evaluation: How to Test AI Outputs

Ask a team how they know their AI feature works and you'll usually get some version of "we tested it." Press on what that means and it comes down to: someone typed in a handful of inputs, read the outputs, and they looked good.

"It looked good" is the most expensive sentence in AI development. It's not a lie — the outputs probably did look good. But it measured nothing, recorded nothing, and covered almost nothing. Two weeks later someone edits the prompt, the outputs shift, and there is no way to say whether things got better or worse — because "good" was never defined and never written down. LLM evaluation is how you replace that sentence with a number.

This is the fourth quiet failure surface in our AI reliability guide — un-measured output — and it's the one that makes the other three unfixable. You can't debug retrieval, catch regressions, or tune an agent if you can't score the result.

One scope note before we start: this post is about evaluating your own system's outputs — the engineering discipline. Choosing which AI vendor or tool to buy is a different problem with different criteria, and it's not what "eval" means here.

An eval has exactly three parts

Strip away the tooling and every eval is the same machine:

1. A test set. Real inputs paired with expected outcomes. Sometimes the expectation is exact ("this support ticket should be classified billing"), sometimes it's a property ("the summary must mention the deadline and stay under 100 words"). Either way, it's written down before the run.
2. A scorer. A function that grades one output against one expectation and returns a score. Scorers range from exact match to rule checks to embedding similarity to an LLM judge — more on the tradeoffs below.
3. A threshold. The line that turns scores into a decision: pass rate at or above the bar, ship; below it, hold. Without a threshold, an eval is a report nobody acts on.

That's it. No platform required, no PhD required. A folder of test cases, a scoring script, and a number your CI can compare against a bar.

Building your first test set

The test set is where most teams stall, because it sounds like a research project. It isn't. The working recipe:

• Start from production, not imagination. Pull real inputs from logs, tickets, and user sessions. Real inputs are messier, shorter, more misspelled, and more hostile than anything you'd invent — which is exactly why they belong in the set.
• Start small and real. A few dozen well-chosen cases beat a thousand synthetic ones. Cover the common cases first, then the edges: ambiguous inputs, empty inputs, adversarial phrasing, and the longest input you legitimately expect.
• Label the expectation, not the exact wording. For generative tasks, define what a passing output must contain, must not contain, and how long it can be. You're grading properties, not prose style.
• Feed it every failure. Each production miss becomes a permanent test case. This is the compounding move: your eval grows exactly where your system is weakest, and a bug fixed once can never quietly return.

Choosing scorers: deterministic first, judge last

The ordering matters. Deterministic scorers — exact match and rules — are reproducible and free, which makes them the foundation you can gate a build on. An LLM judge grading another LLM adds cost and its own variance; use it where subjective quality genuinely matters, and spot-check its verdicts against human judgment before you trust it. A useful default: deterministic checks decide the gate, the judge adds color.

Eval-driven development: the loop that replaces vibes

Once the eval exists, the working rhythm flips. Instead of tune-then-eyeball, you eval-then-tune:

1. Write the eval first. Before touching the prompt, define what passing looks like in test cases. This forces the question "what is this feature supposed to do?" to get a written answer.
2. Baseline the current behavior. Run the eval, record the score. This number is now the thing to beat.
3. Change one thing, rerun, compare. Prompt edit, model swap, parameter change — the eval says whether it helped, hurt, or did nothing. No more arguing from three cherry-picked outputs.
4. Gate the merge. Wire the eval into CI so a change that drops the score below your bar can't ship. This is where evals meet prompt regression testing — the baseline diff that catches what a score summary can miss.
5. Version everything. Prompts get versions, changelogs, and owners, like code — because a prompt with an eval score attached is finally a managed artifact instead of a string someone edits at midnight.

The honest boundary

An eval cannot prove your AI is always right. The test set is a sample, not the universe; the model stays probabilistic; and a pass rate is a measurement, not a warranty. What an eval proves is narrower and far more useful: this change didn't make things worse, this version beats that version on the cases we care about, and quality is trending in a direction you can see. Teams that have that stop debating outputs in Slack and start reading a chart. That's the entire trade — and it's a good one.

Keep going: the full four-surface program is in AI reliability: how to know your AI feature actually works. To put a hard baseline-diff gate behind your eval, read prompt regression testing. The same eval is what you rerun when the provider moves the model rather than when you move the prompt — that case is model deprecation. And if your outputs depend on retrieval, score that layer with its own metrics: RAG evaluation.