How We Evaluate a RAG System Before Shipping It: Building a Real RAGAS Test Harness

Ganesh Sharma
Jul 20
11 min read

Updated: Jul 21

The Question Every RAG Project Eventually Faces

At some point in every retrieval-augmented generation (RAG) project, someone asks the same question: "How do we actually know this is working?" The demo always looks good: a few friendly questions, well-chosen documents, a confident answer. But a demo is not a system, and "it looked right when I tried it" is an anecdote, not an evaluation.

That gap, between a demo that looked good and a system reliable enough for customers or employees, is where most RAG projects quietly stall, shipping something that feels right and then fielding complaints nobody can reproduce or measure. This is the problem a proper evaluation harness solves, and here is how we approach it before anything goes to production.

Why "It Sounds Right" Is Not Good Enough

A RAG system can fail in ways that are easy to miss and expensive to ignore:

It answers confidently using information that is not actually in the retrieved documents: a hallucination dressed up as a citation.
It retrieves the wrong documents entirely, so even a perfectly honest model is reasoning from the wrong material.
It retrieves the right documents but misses the specific paragraph that actually answers the question, so the answer is technically grounded but incomplete.
It answers a question the user did not ask, because the retrieval step drifted toward a related-but-wrong topic.

None of these failure modes are visible from a transcript that merely "reads fine." They require pulling the pipeline apart at each stage, examining what was retrieved and what was generated from it, and scoring each stage independently.

That is precisely what a structured evaluation framework does. We use RAGAS (Retrieval-Augmented Generation Assessment) as the scoring layer, because it gives each stage of the pipeline its own metric instead of one vague "did it seem okay" judgment.

The Three Questions RAGAS Actually Answers

Stripped of jargon, a RAG evaluation harness is answering three separate questions, in this order:

Did we find the right material? (retrieval quality)
Did we find enough of the right material? (retrieval completeness)
Did the model actually stick to that material, or did it wander off and make something up? (generation faithfulness)

RAGAS maps each of these to a distinct, independently-scored metric:

Metric	Plain-English question it answers	What a low score tells you
Context Precision	Of everything the system retrieved, how much was actually relevant?	Your retrieval step is pulling in noise: wrong documents, wrong sections, wasted context window.
Context Recall	Of everything relevant that existed, how much did the system actually find?	Your retrieval step is missing material. The answer may be incomplete even if what it did find was accurate.
Faithfulness	Of everything the model said, how much is actually supported by what it retrieved?	The model is generating claims that are not grounded in the source material. This is where hallucination hides.
Answer Relevancy	Does the generated answer actually address the question that was asked?	The model may be technically accurate but off-topic, verbose, or answering a nearby question instead of the real one.

The reason to separate these is diagnostic. A single overall "quality score" tells you that something is wrong. Four independent scores tell you where, and "where" is what determines whether the fix is a retrieval tuning problem, a prompt problem, or a data problem. Those are three different teams' work, and conflating them wastes weeks.

The flow above has four stages. It starts with the user's question. The question enters the retrieval step, where the system pulls candidate documents from the knowledge base and that step gets scored on context precision and context recall. The retrieved material then enters the generation step, where the model writes an answer from what it was given, and that step gets scored on faithfulness and answer relevancy. Only after both stages pass does the final answer reach the user. Splitting the flow this way is what makes the diagnosis possible. If the retrieval step's scores are healthy but the generation step's are not, the fix lives in the prompt, not in the search index, and vice versa.

How These Scores Actually Get Computed

None of these four metrics come from a human reading every response by hand, and none of them come from a simple keyword match either. RAGAS uses a language model as the judge, but a constrained one, asked narrow yes-or-no questions instead of an open-ended quality rating.

Faithfulness works by decomposition. The generated answer gets broken down into individual factual claims, one sentence or assertion at a time, and each claim is checked against the retrieved context on its own. If three of five claims trace back to the retrieved material and two do not, faithfulness comes back low, and the two unsupported claims are exactly what a reviewer should look at first, not the whole answer.

Context precision works the other direction. Each retrieved chunk gets checked against the reference answer to see whether it was actually relevant, then the metric rewards relevant chunks that rank high, not just relevant chunks that happen to be somewhere in the list. A retrieval step that buries the one useful document under nine irrelevant ones scores worse than one that surfaces it first, even if both technically retrieved it.

Context recall compares the retrieved material against what the golden dataset says the answer actually requires. If the reference answer depends on three distinct facts and the retrieval step only surfaced material covering two of them, recall reflects that gap even if the answer sounds confident.

Answer relevancy works backward from the answer. The system generates a set of questions that the answer would plausibly be responding to, then measures how semantically close those generated questions are to the original question. An answer that wanders onto a related topic produces reverse-engineered questions that drift from what was actually asked, and that drift is the low score.

None of this requires understanding the underlying implementation to use well. What matters for reading a report is simpler: precision and recall are about what got retrieved, faithfulness and relevancy are about what got written, and each is checked in isolation rather than folded into one number.

Step One: The Golden Dataset

Before any scoring happens, we build what we call a golden dataset: a curated set of realistic questions paired with the answer a domain expert would consider correct, and, where relevant, the specific source passages that answer should come from.

This is deliberately not a set of easy softball questions. A useful golden dataset includes:

Common questions the system will face constantly, so baseline reliability is measurable.
Edge-case questions that sit at the boundary of what the knowledge base covers, so we can see where the system should say "I do not know" instead of guessing.
Ambiguous or multi-part questions that require pulling from more than one document, since single-document lookups are the easy case.
Adversarial questions phrased to tempt the model into answering from general knowledge instead of the retrieved material.

The golden dataset is built with the enterprise's own subject-matter experts, not guessed at from the outside. This step is the one most often skipped under deadline pressure, and it is also the single biggest predictor of whether an evaluation is trustworthy or theater. A harness scored against ten easy questions will always look great and will tell you nothing about the questions that actually matter in production.

Where the Questions Actually Come From

The strongest golden datasets are not invented at a whiteboard. They come from real usage: support tickets, sales call transcripts, internal questions from employees, and the actual queries users typed into an earlier version of the system, if one exists. Real questions carry real phrasing quirks, real ambiguity, and real assumptions that a team brainstorming questions in a conference room tends to smooth over without noticing. When no usage history exists yet, domain experts drafting questions should be told explicitly to write the way a confused or rushed real user writes, not the way a textbook states a problem.

Keeping It Current

A golden dataset is not static. As the knowledge base grows, shrinks, or gets corrected, some of the reference answers built against the old material quietly become wrong, and a harness that keeps scoring against a stale answer will eventually reward the wrong behavior. Reviewing the dataset on the same cadence as major content updates, not just once at the start of the project, keeps the harness measuring the system the enterprise actually has today, not the system it had six months ago.

Step Two: Running the Harness

With a golden dataset in hand, the harness itself is mechanically simple, which is the point: it needs to be something the team runs on every change, not a one-time exercise before launch.

For every question in the golden dataset, the harness:

Runs the question through the actual production retrieval and generation pipeline, unmodified.
Captures exactly what was retrieved and exactly what was generated, not a summary but the raw evidence.
Scores that pair against all four metrics.
Rolls the per-question scores up into an aggregate report, broken down by question category (common, edge-case, ambiguous, adversarial) rather than a single blended number.

The loop closes with a fifth step the diagram makes explicit: whatever the aggregate report points to, whether that is retrieval, the prompt, or the source data itself, gets tuned or fixed. The same golden dataset then runs straight back through the harness again. That return arrow is the part teams skip when they treat evaluation as a one-time exercise. Running the dataset again after every fix is what turns "we think that helped" into a measured before-and-after, instead of a guess dressed up as confidence.

Running it against the actual production pipeline, not a simplified test version, matters more than it sounds. Evaluation environments that diverge from production (different chunk sizes, different retrieval settings, a "cleaner" test index) produce scores that look reassuring and mean nothing.

Step Three: Reading the Scores and Acting on Them

A score in isolation is not a decision. What makes an evaluation harness useful is having thresholds tied to actual go or no-go decisions, agreed on before the results come in, not adjusted afterward to fit whatever number the system happened to produce.

In practice, this looks like:

Faithfulness below threshold on adversarial questions: the model is willing to guess when it should not. Fix: tighten the generation prompt's instructions on refusing to answer outside the retrieved context, and consider adding an explicit "insufficient information" response path.
Context Recall below threshold on multi-part questions: the retrieval step is not pulling in enough breadth. Fix: this is a retrieval-tuning problem (chunk size, retrieval count, or query reformulation), not a prompting problem, and should not be handed to whoever owns the prompt.
Context Precision below threshold across the board: the system is retrieving too much irrelevant material, diluting what the model has to work with. Fix: usually a re-ranking step, or tighter similarity thresholds at retrieval time.
Answer Relevancy dropping on ambiguous questions while the other three metrics hold: the model is grounded and honest but not addressing what was actually asked. Fix: this is a generation-prompt and query-understanding issue, separate from retrieval entirely.

The value of scoring each stage independently is exactly this: it turns "the RAG system is bad at answering questions" (a sentence nobody can act on) into "context recall is failing on multi-part questions, which is a retrieval configuration issue," which is a ticket someone can pick up this week.

Common Mistakes That Undermine an Evaluation Harness

A harness that exists is not the same as a harness that works. The same handful of mistakes shows up across teams building their first one.

Testing on too few questions. A ten-question smoke test can confirm the system is not completely broken, but it cannot detect a regression that only shows up on multi-part or adversarial questions, simply because there are not enough of those question types in the sample to move the aggregate score.
Skipping adversarial and edge-case questions. A golden dataset made entirely of questions the system is likely to answer well produces evaluation scores that flatter the system and tell an enterprise nothing about where it actually breaks.
Reporting one blended score instead of a breakdown by category. An aggregate number that mixes common, edge-case, ambiguous, and adversarial questions together can look stable even while performance on the hardest category quietly degrades, because strong performance on the easy majority masks it.
Treating evaluation as a one-time pre-launch audit. A harness run once before launch and never again catches nothing about the regressions introduced by the next six months of prompt tweaks, model upgrades, and document changes. The value is almost entirely in the repetition.
Testing against a cleaner environment than production. A smaller test index, a simplified retrieval configuration, or a hand-picked document set produces scores that do not transfer to what real users actually experience.
Cherry-picking which failures get investigated. When a fix is applied to the specific failing example that prompted it, without checking whether that fix helps or hurts the rest of the golden dataset, teams can improve one visible case while quietly regressing several invisible ones. This is exactly what re-running the full harness after every fix is meant to catch.

Each of these mistakes has the same underlying shape. They make the evaluation easier to pass without making the system more reliable, which defeats the entire purpose of building the harness in the first place.

How Often to Run It

A harness that only runs when someone remembers to run it eventually stops running. The teams that get the most value treat it as part of the release process rather than an optional extra step.

A fast subset of the golden dataset, the ten or twenty questions most likely to catch an obvious regression, can run automatically on every change that touches the prompt, the retrieval configuration, or the document pipeline, giving a result in minutes rather than waiting for a scheduled run. The full golden dataset runs on a slower cadence, nightly or before any release that reaches production, since the complete set can take longer to score and is meant to catch subtler regressions the fast subset would miss.

Any change to the underlying model, the embedding model, or the vector database configuration deserves a full run regardless of the regular schedule, since these are exactly the changes most likely to shift scores in ways nobody predicted. A provider upgrade that quietly changes embedding behavior, for instance, can move context precision and recall without anyone touching the retrieval code at all, and only a scheduled full run would catch it before a customer does.

Why This Matters Before You Ship

An evaluation harness built this way becomes a permanent part of the system, not a one-time audit. Every prompt change, every retrieval tuning pass, every new document source gets run back through the same golden dataset before it goes live. That turns "did this change make things better or worse?" from a guess into a measured answer, in minutes rather than in the following month's support tickets.

For an enterprise deciding whether to build a RAG system in-house, bring in outside engineering support, or buy from a vendor, this is the question worth asking directly: what does your evaluation process actually look like, and can I see a report from it? If the honest answer is "it seemed to work in testing," that is worth knowing before launch, not after.

Who Can Benefit

Enterprise engineering leaders who need proof a RAG system is reliable before it reaches customers or employees.
Product teams shipping RAG features that look good in demos but generate support tickets nobody can reproduce or diagnose.
Enterprises deciding between building RAG in-house, bringing in outside engineering support, or buying from a vendor.
Enterprises already running RAG in production that have never measured it beyond a general impression that it seemed fine.

How Codersarts Can Help

Codersarts builds evaluation into every RAG system we deliver, scaled to your stage.

A proof of concept gets a lightweight golden dataset and a fast validation pass.
An MVP gets a working RAGAS harness against your core user flows, with real launch thresholds in place.
A full-scale deployment gets the complete evaluation pipeline, integrated into your release process and owned by your team with our support behind it.

We also run independent evaluation audits on existing RAG systems, pinpointing exactly where retrieval or generation is underperforming and what needs to be fixed.

Reach out at contact@codersarts.com or visit www.codersarts.com to get started.

Continue Your AI Learning Journey with Codersarts

If you enjoyed this article and would like to discover more about modern AI applications, production-ready LLM systems, and real-world RAG and MCP implementations, be sure to explore these other blogs from Codersarts: