Cutting Through the Noise: How We Took Context Precision from 61% to 94% in a Legal-Tech RAG System

pratibha00
Jul 20
14 min read

A legal-tech platform came to us with a RAG system that was technically "working." Retrieval was fast. The pipeline never crashed. Dashboards were green. Users got answers, and on the surface, nothing looked broken.

The problem showed up only when you looked closely at what the LLM was actually being handed before it generated a response. A third of the retrieved context was wrong — but not obviously wrong. It was adjacent: the right contract, the wrong clause. The right case, an outdated version. The right topic, buried three paragraphs away from the section that actually answered the query. Nothing threw an error. Nothing looked like a bug. It just quietly degraded trust, one query at a time.

For most applications, that kind of near-miss retrieval is an annoyance — a slightly less helpful answer, a minor inefficiency. For a legal product, it's a liability. When the source material is contracts, statutes, or case law, "close enough" context doesn't just produce a worse answer. It produces a wrong one delivered with full confidence, and in this domain, that's the kind of error that erodes user trust fast and doesn't come back easily.

When we measured it properly — using a golden query set and a repeatable evaluation harness rather than eyeballing outputs — context precision (the share of retrieved chunks that were actually relevant to the query) sat at 61%. That number matched what the team had already sensed anecdotally: too many "why did it pull that" moments, too much manual double-checking, too little confidence in the system doing what it was built to do.

After a focused retrieval rebuild — no fine-tuning, no new model training, no swapping the underlying LLM — we brought that number to 94%. Same base model. Same infrastructure budget, roughly. The difference came entirely from how content was chunked, retrieved, ranked, and passed forward.

This post walks through exactly how: the diagnosis process, the specific architecture changes we made, the tradeoffs we accepted along the way, and the results that followed. If you're running a RAG system where "it mostly works" isn't good enough — because your users are lawyers, auditors, or anyone else who can't afford a confidently wrong answer — this is the playbook we used, and the one we'd use again.

Where It Started: A Platform Under Quiet Strain

Our client operates a legal-tech platform used by in-house counsel and law firm associates to research contract terms, precedent, and regulatory language across a large, continuously growing document set. At the time we engaged, the platform was indexing several hundred thousand documents — contracts, amendments, case filings, and regulatory guidance — spanning multiple jurisdictions and, in many cases, multiple versions of the same underlying agreement.

The RAG system sat at the core of the product's value proposition: a user could ask a natural-language question — "What's the termination notice period in the Q3 vendor agreements?" or "Has this indemnification clause changed since the last amendment?" — and get back a synthesized answer grounded in the retrieved source text. When it worked, it saved associates hours of manual document review. When it didn't, it created a different kind of work: verifying whether the system's answer could actually be trusted.

That verification tax was the real problem. Internally, the team had started noticing a pattern in user feedback and support tickets: answers that referenced the wrong version of a contract, citations that were technically on-topic but not actually responsive to the question asked, and — most damaging — a small but steady stream of cases where the retrieved clause looked right but wasn't the one that governed the current agreement. None of this showed up as a system failure. It showed up as declining trust, users manually re-checking source documents "just in case," and a slow drift away from relying on the tool for anything high-stakes.

Leadership had a hunch the retrieval layer was underperforming, but no hard numbers to confirm it or to prioritize a fix against other roadmap items. That's where the engagement started: not with "fix our RAG system," but with "help us find out if our RAG system is actually the problem — and if so, by how much."

Why Legal Text Breaks Naive RAG

Most RAG tutorials are built and tested on relatively forgiving content — blog posts, product docs, wikis. Legal text is a different animal, and a retrieval architecture that performs well on general content will often quietly underperform on legal content without anyone noticing why.

A few characteristics of the document set made this engagement harder than a typical RAG implementation:

Dense, self-referential language. Legal clauses routinely reference other clauses, defined terms, and external statutes within the same sentence. A single paragraph might be unintelligible without the definitions section three pages earlier. Naive chunking — splitting by fixed token counts — regularly severed clauses from the definitions or cross-references they depended on, so a chunk could be retrieved that was technically "about" the right topic but was missing the context needed to interpret it correctly.

High boilerplate similarity. Contracts share enormous amounts of standard language — indemnification clauses, force majeure provisions, termination language — that's nearly identical across hundreds of documents, with the meaningful differences concentrated in a handful of words or a single modified sentence. Standard embedding similarity struggles here: two clauses can be 95% textually similar and still mean very different things legally. This is exactly the condition where vector search alone tends to retrieve confidently wrong matches.

Versioning and amendments. Many agreements existed in multiple versions — original, amended, restated — often with overlapping but not identical language. Without strong metadata handling, retrieval had no reliable way to distinguish "the clause as currently governing" from "the clause as it existed before the 2022 amendment." Both were valid documents in the corpus. Only one was the right answer for most queries.

Structural nesting. Legal documents are heavily hierarchical — sections, subsections, exhibits, schedules — and the meaning of a fragment often depends on where it sits in that structure. Flat chunking strategies discard that hierarchy entirely, treating a top-level definition and a buried sub-clause exception as equivalent, undifferentiated text.

None of these problems are unique to this client. They're characteristic of legal content generally, which is exactly why so many legal-tech RAG deployments plateau at "good enough for low-stakes queries" and struggle to earn trust for anything that actually matters. Any fix that didn't account for these specifics head-on was never going to move the precision number in a meaningful or durable way.

Diagnosing the Problem: Measuring Before Fixing

Before touching any part of the architecture, we needed a real number — not a vibe. "The answers feel off sometimes" isn't something you can improve against, and it isn't something you can prove improvement on later either. So the first two weeks of the engagement went entirely into building an evaluation harness, not writing retrieval code.

Building a golden query set. Working with the client's team, we assembled a representative set of real user queries pulled from support tickets, product analytics, and interviews with associates who used the tool daily. For each query, a subject-matter reviewer identified the ground-truth passages that should be retrieved to answer it correctly — including cases where the honest answer was "the current system can't retrieve this correctly because the needed context spans two documents." This gave us a benchmark grounded in how the product was actually used, not synthetic or generic test queries.

Defining context precision concretely. We measured context precision as the proportion of retrieved chunks, across the top-k results for each query, that a reviewer judged genuinely relevant and responsive to that specific query — not just topically related. This distinction mattered enormously in a legal corpus, where "topically related but not responsive" was the exact failure mode doing the most damage.

Auditing the existing pipeline. With the benchmark in place, we mapped the current architecture end to end:

Chunking: fixed-size token windows (roughly 512 tokens), with no awareness of clause boundaries, section structure, or document hierarchy.
Embedding & retrieval: a single dense embedding model, cosine similarity search, no hybrid or keyword-based retrieval layer.
Ranking: top-k results passed directly to the LLM in similarity order — no reranking step to re-evaluate results against the query's actual intent.
Metadata: minimal use of document metadata (no consistent handling of version, effective date, or document status) in the retrieval logic itself.

Running the golden set against this existing pipeline confirmed the 61% baseline — and, more usefully, showed us where the failures clustered. Precision was noticeably worse on queries involving amended documents, queries requiring cross-referenced definitions, and queries where boilerplate similarity was high. That pattern gave us a prioritized list of what to fix first, rather than a vague mandate to "improve retrieval."

The Fixes: Rebuilding Retrieval Without Touching the Model

With the diagnosis in hand, the work became targeted rather than exploratory. Every change below was chosen because it addressed a specific failure pattern we'd identified in the golden set — not because it's a generically "best practice" worth applying everywhere.

1. Structure-Aware Chunking

Before: Fixed-size chunks of ~512 tokens, applied uniformly regardless of document type or internal structure.

After: Chunking driven by the document's actual structure — clause boundaries, section and subsection headers, and defined-term blocks kept intact rather than split mid-thought. Where a clause depended on a definition elsewhere in the document, we attached that definition as linked context rather than relying on the chunk to stand alone.

Why: This directly addressed the "right topic, missing context" failure mode from Section 3. A chunk boundary that respects legal structure is far more likely to contain a complete, interpretable unit of meaning — which matters more in legal text than in almost any other domain.

2. Hybrid Retrieval (Dense + Sparse)

Before: Dense embedding search only.

After: A hybrid approach combining dense embeddings with a sparse, keyword-based retrieval method (BM25), with results merged and normalized before ranking.

Why: Boilerplate similarity was the clearest justification here. Dense embeddings alone tend to treat near-identical clauses as near-identical matches, even when the one differing sentence is the legally significant part. Sparse retrieval is far better at surfacing exact-term matches — specific defined terms, section numbers, party names — that dense similarity tends to smooth over.

3. A Dedicated Reranking Layer

Before: Top-k results passed to the LLM in raw similarity order.

After: A cross-encoder reranking step applied to the top-N candidates from hybrid retrieval, re-scoring each against the specific query before final selection.

Why: This was the single highest-leverage change. Initial retrieval is optimized for recall — casting a wide net. Reranking is where precision gets recovered, by directly modeling query-passage relevance rather than relying on embedding geometry alone. This is also where we saw the clearest before/after separation in the golden set.

4. Metadata-Aware Filtering and Query Rewriting

Before: No systematic handling of document version, effective date, or status in the retrieval path.

After: Metadata (version, amendment status, effective dates, jurisdiction) was indexed alongside content and used both to filter retrieval candidates and to rewrite ambiguous queries — for example, resolving "the current agreement" to the specific document version actually in effect for a given date or context.

Why: This targeted the versioning failure mode directly. No amount of better text retrieval fixes a problem that's fundamentally about selecting the right document, not the right passage.

5. An Iterative Evaluation Loop

Rather than making all these changes at once and re-measuring at the end, we ran the golden set after each change in isolation, which let us attribute precision gains to specific interventions and catch regressions early — including one case where an early version of the reranker slightly hurt precision on multi-document queries, caught before it shipped.

What we deliberately didn't do: fine-tune the embedding model, train a custom retriever, or change the underlying LLM. Every gain came from architecture and pipeline design around the existing model stack — which matters for reproducibility, cost, and how quickly this kind of engagement can realistically move the needle for other teams in a similar position.

The Results: What Actually Moved

After implementing the changes listed above and re-running the golden query set, context precision rose from 61% to 94% — a 33-point improvement, achieved entirely through retrieval architecture changes, with no fine-tuning and no change to the underlying LLM.

But precision alone doesn't tell the full story, so we tracked a set of supporting metrics to understand the tradeoffs and confirm the improvement was real rather than a benchmark artifact.

Metric	Before	After
Context precision	61%	94%
Precision on amended-document queries	42%	91%
Precision on high-boilerplate-similarity queries	48%	89%
Average retrieval latency	~180ms	~240ms
Manual verification rate (user self-reported)	High	Substantially reduced

A few things worth calling out in these numbers:

The gains weren't evenly distributed — and that's a good sign. The biggest improvements came exactly where we predicted they would: amended-document queries and high-boilerplate-similarity queries, the two failure modes we'd identified as most damaging in the diagnosis phase. That alignment between predicted and actual improvement is what tells you the fix addressed the real problem, rather than just moving the average through unrelated gains.

Latency increased, and we accepted that tradeoff deliberately. Adding a reranking step and hybrid retrieval merge adds computation. The roughly 60ms increase was evaluated against the cost of the alternative — users manually re-verifying answers, or worse, trusting a wrong one — and was an easy tradeoff to accept for a legal product where correctness matters more than shaving milliseconds off response time.

We validated on a held-out set, not just the golden set used for tuning. To guard against overfitting our fixes to the exact queries we'd been testing against, we ran a second, held-out batch of queries the team hadn't seen during development. Precision on that held-out set came in within two points of the golden-set result, which gave us confidence the improvement would hold up in production rather than just on paper.

Downstream effects showed up beyond the metric itself. While context precision was the primary target, the client also reported a meaningful drop in support tickets related to "wrong" or "outdated" answers in the weeks following deployment, along with qualitative feedback from associates that they were spending less time double-checking retrieved clauses against source documents — the exact verification tax described above.

The number that matters most for a blog headline is 61% → 94%. The number that matters most for the client's business is what that translated to: a system associates could actually rely on, instead of one they had to work around.

What This Meant for the Business

Metrics convince technical stakeholders. What convinces everyone else is what changed in the day-to-day experience of using the product — and for this client, that shift showed up in three concrete ways.

Trust came back. Before the rebuild, associates had developed a quiet workaround: treat the tool's answers as a starting point, then manually verify against source documents before relying on anything. That habit was rational given the 61% baseline, but it also meant the product wasn't delivering on its core promise. Post-rebuild, the client reported associates increasingly citing the tool's output directly, without the reflexive double-check — the clearest sign that trust, not just accuracy, had been restored.

Time saved became measurable, not anecdotal. The verification tax described earlier wasn't just a vague frustration — it was hours per week, per associate, spent re-checking retrieved clauses that should have been trustworthy the first time. With precision at 94%, that overhead dropped sharply enough that the client could point to it as a concrete efficiency gain when talking to their own customers and stakeholders, rather than a qualitative "it feels better" claim.

Support burden shrank. Fewer wrong or outdated answers meant fewer tickets about wrong or outdated answers — freeing up the client's support and product teams to focus on feature requests and genuine edge cases, rather than fielding a steady stream of "why did it show me this" complaints that were really retrieval problems in disguise.

And perhaps most importantly for a legal product specifically: the risk profile changed. In most software categories, an occasional wrong answer is an inconvenience. In legal research, a confidently wrong answer carries real downstream risk — a missed termination deadline, an outdated clause treated as current, a citation that doesn't actually govern.

Closing that gap wasn't just a UX improvement. It was risk reduction that the client could stand behind when talking to their own enterprise customers about why they could trust the platform with higher-stakes work.

That's the throughline worth remembering: context precision is a retrieval metric, but what it actually buys a legal-tech product is permission to be trusted with more important questions.

What We'd Tell Other Teams

This engagement wasn't unusual because the problems were exotic — it was unusual because the client took the time to measure precisely before reaching for a fix. That's the biggest transferable lesson, but a few others are worth calling out for any team running a RAG system in a high-stakes domain.

Measure before you architect. It's tempting to jump straight to "let's add reranking" or "let's try hybrid search" based on general best practices. Those changes worked here because the diagnosis told us exactly which failure modes to target — amended documents, boilerplate similarity, missing cross-references. Without that diagnosis, the same fixes could easily have been applied in the wrong order, or with the wrong emphasis, and produced a smaller, less durable improvement.

Your embedding model is probably not the bottleneck. It's a common instinct to blame the embedding model when retrieval underperforms, and the common response is to consider fine-tuning or swapping it out. In this engagement, the embedding model was never the primary problem — chunking, ranking, and metadata handling were. Before investing in model-level changes, it's worth confirming the pipeline around the model is actually giving it a fair chance to succeed.

Domain structure is a retrieval signal, not just formatting. Legal documents (like many other specialized domains — medical records, financial filings, technical standards) have structure that carries real semantic meaning: hierarchy, cross-references, defined terms, versioning. Treating that structure as retrieval metadata, rather than something to strip out during preprocessing, was one of the highest-leverage decisions in this project.

Reranking earns its latency cost more often than teams expect. The instinct to avoid reranking for speed reasons is understandable, but in domains where a wrong answer is costlier than a slow one, the tradeoff usually favors adding the step. The right question isn't "does this slow us down" — it's "what does an unreranked wrong answer cost us instead."

Precision problems are often solvable without retraining. Retraining or fine-tuning is a real lever, but it's an expensive and slow one, and it's not always the right first move. In this case, architecture-level changes to chunking, retrieval, and ranking closed the gap entirely — which matters for any team trying to improve their system without a lengthy model development cycle standing between them and better results.

Is This You?

Context precision problems rarely announce themselves as a single obvious failure. More often, they show up as a pattern of smaller signals that get written off individually — a support ticket here, a shrug from a user there — until you add them up and realize they're all pointing at the same root cause.

If several of these sound familiar, there's a good chance your RAG system has a precision problem worth measuring:

Users manually double-check retrieved answers before trusting them, especially for anything that matters. If your product's power users have quietly developed a "verify before you rely on it" habit, that's a trust signal, not a training issue.
Retrieved content is topically right but not actually responsive. The system pulls something about the right subject, but not the specific passage that answers the question asked. This is one of the hardest failure modes to catch just by spot-checking outputs, because the answers often look reasonable on the surface.

Your documents have versions, amendments, or supersession logic, and you're not confident retrieval is consistently distinguishing "current" from "superseded." This is a common blind spot in any domain with document lifecycles — legal, compliance, policy, finance.

Your corpus has a lot of near-duplicate or boilerplate content, and you suspect (or have seen evidence) that retrieval sometimes surfaces the wrong instance of a very similar passage.

You've never actually measured context precision — you have a sense that retrieval "seems fine" or "seems a little off," but no benchmark, golden set, or repeatable evaluation to confirm it either way.

You're considering fine-tuning or swapping your embedding model as a fix for retrieval quality issues, but haven't ruled out chunking, ranking, or metadata handling as the actual bottleneck first.

None of these are unusual problems. They're the default state of most RAG systems that were shipped to hit a deadline rather than tuned against a real benchmark. The good news, based on this engagement, is that they're also usually fixable without the cost or timeline of a retraining project.

Where to Start

The gap between "our RAG system mostly works" and "our RAG system is something we'd stake a client relationship on" usually isn't a model problem. It's a retrieval problem — and as this engagement showed, it's one that can be diagnosed and closed without retraining, without swapping your LLM, and without a multi-month roadmap item.

The first step isn't a rebuild. It's a measurement.

If you're not sure where your own context precision actually stands, that's the right place to begin — not with an assumption, but with a number. We help teams build a golden query set from their real usage patterns, benchmark current context precision, and identify the specific failure modes worth fixing first — the same process that uncovered the 61% baseline in this case study.

No commitment to a full rebuild required. Just a clear, evidence-based answer to the question every team running a production RAG system should be able to answer and usually can't: how good is our retrieval, really?

If that's a question you'd rather answer with data than guesswork, let's talk. Reach out and we'll walk through what a retrieval audit would look like for your system.