RAG Evaluation: How to Grade Your Retrieval

Here's the uncomfortable truth about most RAG problems: the model didn't hallucinate. It answered exactly the question you asked, from exactly the context you gave it. The context was just wrong. Retrieval pulled three chunks about the 2023 pricing page when the user asked about 2025, and the model — doing its job — wrote a fluent answer from stale facts.

This is why RAG evaluation starts at retrieval, not at the answer. The generation step gets all the attention, but it sits downstream of a search problem. If the right chunk never reaches the model, no prompt engineering, no better model, and no amount of "please be accurate" can recover it. Garbage in, confident garbage out.

It's also one of the four quiet failure surfaces we map in the AI reliability pillar guide — quiet because nothing errors. Retrieval always returns something. The question is whether it returned the right thing, and most teams have no way to answer that.

Why RAG feels right but retrieves wrong

Vector search is similarity search. It returns the chunks whose embeddings sit closest to the question's embedding — which is often, but not always, the chunk that answers the question. The gaps are predictable:

• Near-miss chunks. A chunk that talks about the topic outranks the chunk that answers the question. Similar isn't the same as relevant.
• Split answers. Chunking cut the key fact in half, so no single chunk contains the whole answer and neither half ranks well.
• Noisy top-k. The right chunk is there — at position 7 — buried under six loosely related ones that crowd the context window and dilute the model's attention.
• Silent drift. You swap an embedding model, tweak chunk size, or re-index new documents, and retrieval quality shifts. Nothing fails. Nobody notices until users do.

Every one of these produces a demo that feels fine. You test the questions you'd naturally ask — usually the easy ones the index handles well — and conclude retrieval works. The misses live in the questions you didn't try.

Precision and recall, in plain language

Retrieval quality comes down to two questions, and the standard metrics are just precise ways of asking them.

Recall asks: did the right chunks show up? Of the chunks that should have been retrieved for this question, how many actually appeared in the top k results? Low recall is the killer — it means the answer never reached the model at all. This is the headline metric.

Precision asks: how much junk came with them? Of the chunks you retrieved, what share were actually relevant? Low precision floods the context window with noise, which costs tokens and drags answer quality down even when the right chunk is present.

You don't need to memorize the formulas. You need the habit: score all of them against a gold set, watch Recall@k first, and treat a drop as a build-breaking event.

Gold sets: the part everyone skips

A gold set is the ground truth that makes grading possible: a list of real questions, each paired with the chunk or chunks that genuinely answer it. Without one, "is retrieval good?" is a matter of opinion. With one, it's arithmetic.

Teams skip this step because it sounds like a labeling project. It doesn't have to be. A useful starter set is small — a few dozen questions — and grows over time:

1. Start with real questions. Pull them from support tickets, chat logs, and search queries — the things users actually ask, phrased the way they actually ask them.
2. Label the answering chunks. For each question, find the passage in your corpus that answers it. That pairing is one gold row. (The RAG Retrieval Grader ships a gold-set template, a labeling workflow, and a helper that bootstraps a starter set from your existing documents, so you're not starting from a blank page.)
3. Add every production miss. Each time a user catches a bad answer, trace it to the retrieval miss and add that question to the gold set. Your test set grows exactly where your system is weakest.

Grading retrieval and gating it in CI

Once the gold set exists, the workflow is short. Run every gold question through your retriever. Score the results — Recall@k, Precision@k, MRR, nDCG. Compare against your thresholds and get a verdict: ship, hold, or fix. Then wire that same run into CI, so an embedding swap, a chunking tweak, or a re-index that drops retrieval below your bar fails the build instead of shipping quietly.

The verdict matters more than the dashboard. A number on a chart invites debate; a failed build forces a decision. Set a minimum Recall@k you're willing to live with, and let the gate hold the line while you experiment freely above it. This is the same baseline-and-gate discipline that prompt regression testing applies to prompts — applied to search. It's also what you rerun when the change wasn't yours at all: an embedding-model update moves retrieval the same way a deprecated or repointed model moves generation.

And when the grade comes back poor? The fix list is usually short: chunking strategy, embedding model, k value, or index hygiene. The point of measuring first is that you change one thing, re-grade, and know — instead of changing five things and hoping.

The honest boundary

A clean retrieval score does not certify your answers. The model can still misread a correct chunk, and no metric here catches a prompt that mangles good context — that's what output-level LLM evaluation is for. What retrieval grading proves is narrower and foundational: the model had the right material in front of it. Prove that first. It's the failure surface most RAG teams have never measured, and the root cause of most of what gets blamed on the model.

Keep going: retrieval is one of four quiet failure surfaces. The full map is in AI reliability: how to know your AI feature actually works. If your pipeline passes retrieval but you still can't say whether v2 beats v1, the next read is LLM evaluation: how to test AI outputs.