What Vendors Won't Tell You: A Framework for Evaluating a RAG System's Real Cost, Latency, and Accuracy

Ganesh Sharma
Jul 20
11 min read

Updated: Jul 21

Every Vendor Deck Looks the Same

If you have sat through more than two vendor pitches for a retrieval-augmented generation (RAG) system, you have likely noticed a pattern. The demo is fast, the answers are accurate, and the pricing slide shows one clean number. Then you sign the contract, and three things happen that were never in the deck: the bill runs three to five times higher, latency is nothing like the demo, and accuracy on your real questions falls short of what was promised.

This is not usually dishonesty, it is that cost, latency, and accuracy can each be measured a dozen different ways, and vendors report whichever version looks best. The fix is not distrust, it is a framework for asking the right follow-up question, and this post lays it out across the three dimensions that actually determine whether a RAG deployment succeeds.

The three pillars above are not three unrelated checks, they are three separate places a vendor's headline number can hide the real story. Cost hides in the four line items nobody quotes alongside the LLM call. Latency hides in the gap between a demo running on a warm cache and a production system under real concurrent load. Accuracy hides in whose questions were used to measure it. Each pillar gets its own section below, in that order, with the specific follow-up question that closes the gap.

Pillar One: Cost, and Why the Number on the Slide Is Never the Whole Number

The quoted price for a RAG system is almost always a per-query or per-seat number for the language model call. That is one line item in a bill that has at least five.

What the full cost stack actually looks like

Cost Component	What Drives It	Why It Is Easy to Undercount
LLM inference	Tokens in (prompt and retrieved context) and tokens out (the answer)	The quoted number usually assumes a short prompt. Real retrieved context is often 3 to 10 times larger than the demo's.
Embedding generation	Every document indexed, and every query at runtime	One-time indexing cost is visible. The runtime embedding cost on every user query is often left out of the quote entirely.
Vector database hosting	Index size, query volume, uptime tier	Scales with your actual document volume, not the vendor's demo corpus. This is frequently quoted at a starter tier that will not hold your real data.
Re-ranking / retrieval tooling	Whether a re-ranking step is used to improve precision	Often an optional add-on priced separately, but frequently necessary in practice to hit accuracy targets.
Ongoing maintenance	Re-indexing as source documents change, prompt tuning, monitoring	Rarely priced at all in an initial quote. This is where the bill grows quietly over the first six months.

The question to ask

Do not ask "what does this cost per query?" Ask instead: "Walk me through every cost line item for 10,000 queries per month against a document set the size of ours, including embedding, hosting, and maintenance, not just the LLM call."

If a vendor cannot answer that breakdown specifically, that is not a red flag about dishonesty. It is a sign they have not run the numbers on a deployment your enterprise's size either, which is arguably more concerning.

Pillar Two: Latency, and Why the Demo Number Is Not the Production Number

A live demo answering one question, with a warm cache and a small, curated document set, tells your enterprise almost nothing about what a real user will experience under real load.

The bar above shows why a single quoted latency number is rarely the whole picture. Each segment is a step the request has to pass through in order, and the width of each segment is roughly how much time it tends to take.

Query embedding and vector search are usually short. Re-ranking, when a vendor uses it, adds a further short segment that improves accuracy but is easy to leave out of a demo. Prompt assembly and generation is typically the widest segment, the largest single chunk of total time. Response delivery closes the request.

A vendor quoting only the generation step's latency is quoting one segment out of five, and the other four still show up in what the user actually experiences.

What actually adds up

A single RAG response is not one operation. It is a chain of at least four to five sequential steps, and the total latency is the sum of all of them, not just the LLM call:

Query embedding: converting the user's question into a vector.
Vector search: retrieving candidate documents from the index. This grows with index size unless the infrastructure is tuned for it.
Re-ranking (if used): a second, more precise pass over the retrieved candidates, which meaningfully improves accuracy but adds a real time cost.
Prompt assembly and LLM generation: usually the largest single chunk of total latency, and highly sensitive to how much context was retrieved.
Streaming vs. full-response delivery: whether the user sees the answer appear progressively or waits for the complete response, which changes perceived latency even when actual latency is identical.

Why the demo number is misleading

Vendor demos are typically run against small indexes, with no concurrent load, and often with a re-ranking step disabled to keep the response snappy. Your production deployment will have a larger index, real concurrent traffic, and, if accuracy matters, which it should, a re-ranking step that the demo skipped.

The question to ask

Ask for p50 and p95 latency figures, not an average, measured against an index sized like your real document set, under realistic concurrent load, with every accuracy-improving step (like re-ranking) turned on. The gap between p50 and p95 tells your enterprise how consistent the system is. A wide gap means some fraction of your users will have a noticeably worse experience than the number on the slide suggests.

Pillar Three: Accuracy, Where "95% Accurate" Is a Claim That Needs a Source

This is the metric most likely to be quietly inflated, not through fabrication, but through favorable measurement conditions. "Our system is 95% accurate" is meaningless without knowing: accurate on what questions, measured how, by whom.

Three questions that separate a real accuracy claim from a marketing one

1. Whose questions were used to measure it?A vendor's own curated test set is, by construction, made of questions their system handles well. Ask whether the accuracy figure was measured against your domain's real questions, including the awkward, ambiguous, and edge-case ones your actual users will ask, or against a benchmark the vendor selected.

2. What, specifically, was scored?"Accuracy" can mean the final answer was correct, or it can mean the system merely retrieved a relevant document, which is a much lower bar. A rigorous evaluation separates these into distinct metrics: did retrieval find the right material, and separately, did the generated answer actually stick to that material without adding unsupported claims. A vendor quoting one blended "accuracy" number is very likely quoting the more flattering of the two.

3. Can your enterprise see the raw evaluation, not just the summary score?A trustworthy accuracy claim comes with a reproducible evaluation report: the specific test questions, what was retrieved for each, what was generated, and how each was scored. A summary slide with a single percentage and no methodology behind it is not evidence, it is an assertion.

The ladder above has three rungs, and most vendor pitches stop on the bottom one. The bottom rung is a bare percentage with no source attached, such as "we are 95% accurate," which carries no way to check it. The middle rung adds a visible test set and a stated methodology, so your enterprise can at least see how the number was produced, even if it was produced on the vendor's own chosen questions. The top rung is the vendor's system run against your enterprise's own domain questions, scored by your own experts. Only the top rung is independently verifiable, which is why it is the one worth insisting on before signing.

The question to ask

Request that the vendor run their system against a small set of your enterprise's own real questions, with your own domain experts scoring the results, not a demo on their chosen material. Almost any credible vendor will agree to this if their numbers are real. Hesitation here is the single most informative signal in the entire procurement process.

Data Privacy and Security

Cost, latency, and accuracy are the three pillars vendors are most often evaluated on, but for a RAG system built on an enterprise's own documents, data handling deserves its own line of questioning, since the documents flowing through the system are frequently the enterprise's most sensitive material.

Ask directly where document content and query logs are stored, and for how long. Ask whether your enterprise's data is ever used to train or fine-tune the vendor's underlying models, since a default opt-in to model training is common and not always disclosed upfront. Ask how the vendor isolates your enterprise's data from other customers in a multi-tenant system, and what happens to your data and embeddings if the contract ends. Ask what compliance certifications the vendor actually holds.

Ask where the underlying infrastructure is physically hosted, and whether the data residency requirements your enterprise operates under, for a specific region or industry, are actually met rather than assumed. Ask whether the vendor can produce an audit log of who accessed which documents and when, since that becomes relevant the moment a security review or compliance audit asks your enterprise the same question about a system it does not fully control.

None of these questions require a security team to ask. They require the same posture as the cost, latency, and accuracy questions above: asking for something specific and verifiable instead of accepting a general assurance.

Red Flags to Watch For During a Vendor Demo

A demo is a controlled environment, and controlled environments hide exactly the things this framework asks about. A handful of patterns are worth watching for regardless of how polished the presentation is.

The same three example queries every time. A demo rehearsed on a fixed, small set of questions says nothing about how the system handles the long tail of real usage. Ask to type a question yourself, live, on a topic the presenter has not prepared for.
Vague answers about the retrieval and re-ranking approach. A vendor who cannot describe, at a reasonable level of detail, how documents get chunked, embedded, and re-ranked is either using an off-the-shelf pipeline they have not customized for your use case, or does not want to discuss the limitations of what they built.
No willingness to discuss pricing tiers in detail. A vendor who cannot walk through what happens to the bill as usage scales, or who insists on a call before sharing any pricing structure at all, is often protecting a number they know will not survive the comparison this framework encourages.
No SLA on latency or uptime. A vendor confident in their production performance will commit to a number in writing. A vendor who will only say the system is "usually fast" is describing the demo, not a guarantee.

None of these signs alone disqualifies a vendor. Together, they indicate how much of the sales conversation was optimized for the demo rather than for your enterprise's actual deployment, which is precisely the gap this framework exists to close.

How to Run a Fair Bake-Off Between Multiple Vendors

Comparing two or three vendors side by side is where this framework earns its keep, but a bake-off run unfairly produces a comparison that looks rigorous while actually just reflecting whichever vendor prepared the best demo.

Use the same test questions for every vendor, drawn from your own golden dataset rather than letting each vendor propose their own showcase scenario. A vendor choosing their own test questions is, understandably, choosing the questions their system handles best, which defeats the purpose of a comparison. Use the same document set for every vendor as well, ideally a real slice of your enterprise's actual content rather than a generic sample, since retrieval quality varies significantly with document structure and vendors can differ sharply on messy real-world documents even when they look identical on clean ones.

Score every vendor's output the same way, ideally with the same reviewers scoring blind, without knowing which response came from which vendor, since knowing the source introduces bias even among reviewers trying to be objective. Measure latency under comparable load for every vendor rather than accepting one vendor's self-reported number alongside another vendor's number measured live, since the two are rarely produced under the same conditions.

Run the cost comparison against the same projected query volume and document size for every vendor, using the line-item breakdown from Pillar One rather than the headline price each vendor quotes. Two vendors quoting the same per-query price can differ by a wide margin once embedding, hosting, and re-ranking costs are added at your actual scale, and that difference only shows up when the full stack is compared, not the headline number.

What a Good Vendor Response Actually Looks Like

The red flags above describe what to watch for, but it is worth being just as specific about what a strong response looks like, since a fair evaluation should be able to reward a vendor that does this well, not just penalize the ones that do not.

A vendor confident in their cost structure walks through the full breakdown unprompted, including embedding and hosting costs, before your enterprise has to ask for it directly. A vendor confident in their latency offers p95 figures under a load comparable to your expected usage, and is willing to have that number verified independently rather than only demonstrated live.

A vendor confident in their accuracy invites your enterprise to bring its own questions and its own domain experts to score the result, rather than steering the conversation back to their own benchmark.

A vendor serious about enterprise data handling has clear, specific, written answers about storage, training use, and compliance certifications ready before the question is even asked, because they have answered it many times before for other enterprise customers.

None of these signals guarantee the system itself is the right fit for your use case, since a transparent vendor can still be the wrong technical match. What they do indicate is a vendor whose other claims are more likely to hold up under the same scrutiny, since transparency about the parts that are easy to verify is a reasonable signal about honesty on the parts that are harder to check independently.

Putting the Framework to Work

None of these three pillars, on their own, tells your enterprise whether to buy. Together, they replace a vendor's summary slide with a set of specific, answerable questions:

Cost: a full line-item breakdown for your actual query volume and document size, not a headline number.
Latency: p50/p95 figures under realistic load and index size, with accuracy-improving steps like re-ranking left on.
Accuracy: a reproducible evaluation, ideally run against your own domain's questions, with retrieval quality and generation faithfulness reported separately.

A vendor confident in their system will have straightforward answers to all three. A vendor who only has a polished demo and a single headline number for each is not necessarily misleading your enterprise, but they have not yet done the work of measuring what actually matters for a deployment your size, and that is worth knowing before the contract is signed rather than after.

Who Can Benefit

Enterprise procurement and technical evaluators comparing multiple RAG vendors who need a structured way to see past the pitch decks.
Enterprises already burned by a quoted cost or accuracy number that did not hold up in production.
Technical leads asked to sign off on a vendor selection who want an independent framework, not just a sales conversation.
Enterprises deciding between building RAG in-house and buying from a vendor, who need real numbers from both sides.

How Codersarts Can Help

We are not selling a RAG platform, so we have no stake in which vendor you pick, which is why enterprises bring us in during procurement. We scale to your stage.

At proof of concept, we validate a vendor's or a custom build's claims quickly, before further budget commits.
At MVP, we set realistic cost, latency, and accuracy targets for the first release.
At full-scale deployment, we build the ongoing monitoring that keeps a vendor, or your own system, accountable long after signing.

Reach out at contact@codersarts.com or visit www.codersarts.com to get started.

Continue Your AI Learning Journey with Codersarts

If you enjoyed this article and would like to discover more about modern AI applications, production-ready LLM systems, and real-world RAG and MCP implementations, be sure to explore these other blogs from Codersarts: