How to Evaluate RAG Quality with Amazon Bedrock: An Enterprise Measurement Guide for 2026

pranavsankar
3 days ago
26 min read

A RAG assistant can answer ten demonstration questions correctly and still be unsafe to release.

The demo may contain only easy factual lookups. The evaluators may already know which documents to search. No one may test expired policies, ambiguous acronyms, unauthorized documents, questions with no answer, or requests that require evidence from several sources. A fluent response can hide a retrieval failure; a correct response can be produced from the model's memory rather than the company's evidence; and a high average score can conceal complete failure for one business-critical category.

That is why RAG quality is not one number and why “the answers looked good” is not a release criterion.

Amazon Bedrock now provides managed RAG evaluation jobs for both Amazon Bedrock Knowledge Bases and externally produced RAG outputs. These jobs are useful, but they are one part of an enterprise measurement system. A production decision still requires deterministic retrieval tests, access-control tests, calibrated human review, operational service levels, cost analysis, and failure-level inspection.

This guide shows how to build that system.

The Short Answer

To evaluate RAG quality with Amazon Bedrock:

Define the decisions the evaluation must support and the failures that matter.
Build a versioned dataset from real query patterns, authoritative answers, expected evidence, and access personas.
Run retrieval-only tests before testing generated responses.
Measure deterministic ranking metrics such as Recall@K, MRR, nDCG, and unauthorized-retrieval rate alongside Bedrock's context relevance and context coverage.
Run retrieve-and-generate evaluation for correctness, completeness, helpfulness, logical coherence, faithfulness, citation precision, citation coverage, harmfulness, stereotyping, and refusal.
Add custom metrics for domain requirements that built-in metrics do not represent.
Calibrate LLM-as-a-judge scores against expert human decisions.
Define release gates by query segment and severity—not just one corpus-wide average.
Compare one controlled system change at a time.
Continue evaluating sampled production traffic, drift, latency, cost, and incidents after launch.

The governing principle is:

Evaluate the component you changed, preserve every relevant version, and inspect the failures behind every aggregate score.

RAG Quality Is a System Property

Retrieval-Augmented Generation has at least five quality surfaces:

Surface	Question	Typical failure
Corpus and ingestion	Is the right knowledge present, current, parsed, and attributable?	The current policy table was lost during parsing
Retrieval and ranking	Did the system find the best authorized evidence?	A related but obsolete document ranked above the controlling policy
Context assembly	Did the model receive sufficient, non-conflicting evidence?	Relevant chunks were retrieved but trimmed from the prompt
Response generation	Is the answer correct, complete, faithful, useful, and appropriately uncertain?	The model invented a condition that was not in the evidence
Product and operations	Is the system secure, fast, affordable, observable, and usable?	Quality passes offline, but p95 latency and authorization failures make the product unacceptable

An end-to-end score cannot reliably tell the team which surface failed. If the final answer is wrong, the cause may be:

Missing or stale source material.
Incorrect parsing or chunking.
A weak embedding representation.
An inappropriate search mode.
Missing metadata filters.
Poor ranking or reranking.
Too few or too many retrieved results.
Context truncation.
A generation prompt that encourages guessing.
A generator model that cannot reason over the evidence.
Citation mapping defects.
Access-control leakage.
A question that should have been refused.

This is why our RAG accuracy methodology evaluates pipeline stages independently. Amazon Bedrock's two RAG evaluation modes fit naturally into that operating model:

Retrieve only: assess the retrieved texts.
Retrieve and generate: assess the retrieved evidence and the generated response.

AWS supports both Amazon Bedrock Knowledge Bases and precomputed inference responses from another RAG implementation. This makes the managed evaluator useful as a common judging layer during a Bedrock Knowledge Bases versus custom RAG comparison.

What Amazon Bedrock Can Evaluate in 2026

Amazon Bedrock Evaluations provides managed, LLM-as-a-judge RAG evaluation jobs. The evaluation uses a prompt dataset in Amazon S3, invokes a supported evaluator model, and writes results to an S3 output location. For a Bedrock Knowledge Base, the service can perform retrieval or retrieve-and-generate calls as part of the job. For an external or custom RAG system, the team supplies precomputed retrieved passages and, for end-to-end evaluation, the generated response.

Two Job Types

Evaluation type	What Bedrock evaluates	Best use
Retrieve only	Retrieved passages for each query	Chunking, embeddings, search, filters, K, reranking, ingestion changes
Retrieve and generate	Retrieved passages plus generated answer	Generator model, prompt, evidence use, citations, response behavior

Run retrieve-only first when retrieval is uncertain. Evaluating a polished response produced from bad evidence can create misleading diagnoses. Run retrieve-and-generate after retrieval clears its minimum gates, or when testing a change that directly affects generation.

Built-In Retrieval Metrics

Amazon Bedrock currently exposes two built-in metrics for retrieve-only RAG evaluation:

Bedrock metric	Meaning	Direction	Important dependency
Builtin.ContextRelevance	How relevant the retrieved text is to the query	Higher is generally better	Does not prove all required evidence was found
Builtin.ContextCoverage	How much retrieved text covers the expected answer information	Higher is generally better	Requires a ground-truth reference response

Context relevance is closest to a noise measure: did retrieval bring back material related to the question? Context coverage is closer to evidence sufficiency: did retrieval cover the information represented in the ground truth?

Neither replaces deterministic document-level ranking metrics when the team knows which source or passage should be retrieved. An LLM judge may reasonably consider two passages semantically equivalent, while an auditor may require the controlling policy version by exact identifier.

Built-In Retrieve-and-Generate Metrics

Amazon Bedrock currently documents ten built-in metrics:

Bedrock metric	What it asks	Preferred direction
Builtin.Correctness	Is the response accurate for the question?	High
Builtin.Completeness	Does it resolve all parts of the question?	High
Builtin.Helpfulness	Is it useful overall?	High
Builtin.LogicalCoherence	Is it free from logical gaps and contradictions?	High
Builtin.Faithfulness	Does it avoid claims unsupported by retrieved text?	High
Builtin.CitationPrecision	Are cited passages cited correctly?	High
Builtin.CitationCoverage	Are response claims adequately supported by citations?	High
Builtin.Harmfulness	How much harmful content appears?	Low
Builtin.Stereotyping	How much generalized stereotyping appears?	Low
Builtin.Refusal	How evasive is the response?	Context-dependent; usually low for answerable queries

The polarity matters. AWS describes every result as a value between 0 and 1, where a value closer to 1 means more of that metric's characteristic is present. A high faithfulness score is favorable; a high harmfulness score is not. A dashboard that simply colors every high value green will invert the safety interpretation.

AWS recommends using citation precision and citation coverage together. Precision without coverage can reward a response that cites one claim correctly while leaving five claims unsupported. Coverage without precision can reward abundant but incorrect citations.

Custom Metrics

Built-in metrics do not know the organization's rules. Bedrock supports up to ten custom metrics in one RAG evaluation job. A custom metric can represent requirements such as:

Uses the controlling policy rather than an expired version.
States jurisdiction-specific qualifications.
Does not offer financial or legal conclusions outside scope.
Uses the mandated answer format.
Includes escalation language for a high-risk condition.
Distinguishes contractual obligation from internal guidance.
Names the effective date when the source contains one.
Refuses to infer a customer's eligibility from incomplete evidence.

The custom metric definition includes evaluator instructions and should include an explicit rating scale. AWS warns that without a rating scale it may not parse results reliably for charts or averages. The metric definition is also written to the evaluation output path, which helps preserve evaluation lineage.

Review the live RAG evaluation metric catalog, because supported evaluators, generators, Regions, and metric behavior change.

Bedrock Metrics Are Necessary, but Not Sufficient

Managed LLM judging is valuable for semantic qualities that are expensive to express with exact rules. It is not the complete quality program.

Add Deterministic Retrieval Metrics

If each test query has a set of expected documents or passages, calculate:

Recall@K: proportion of relevant evidence found in the first K results.
Precision@K: proportion of the first K results that is relevant.
Hit rate@K: whether at least one expected result appears in the first K.
MRR: reciprocal rank of the first relevant result, averaged across queries.
nDCG@K: ranking quality when relevance has graded levels.
Exact source-version hit rate: whether the authoritative version appeared.
Duplicate-context rate: how much of the context repeats substantially identical content.
Retrieval abstention accuracy: whether the system returns no answer when no authorized evidence exists.

For a set of queries (Q), a simple Recall@K definition is:

Recall@K = average over q in Q of
           |relevant(q) intersect top_k(q)| / |relevant(q)|

MRR focuses on how soon the first relevant result appears:

MRR = average over q in Q of 1 / rank_of_first_relevant_result(q)

These metrics are transparent and reproducible. They also reveal failure patterns an LLM-based relevance score may blur.

Add Non-Negotiable Security Metrics

Authorization is not a subjective quality dimension. Measure it deterministically:

Unauthorized document retrieval rate.
Unauthorized citation rate.
Cross-tenant evidence rate.
Revoked-access propagation time.
Metadata-filter enforcement rate.
Prompt-injection success rate from retrieved content.
Sensitive-data exposure rate.

The acceptable rate for a cross-tenant retrieval test is usually zero, not an average above 0.9. If one persona can retrieve another tenant's source, a favorable helpfulness score does not offset the breach.

Add Operational Metrics

Track the user experience and economics:

Retrieval latency p50, p95, and p99.
End-to-end time to first token and completion.
Timeout, throttle, retry, and error rates.
Tokens and retrieved characters per answer.
Retrieval, reranking, generation, Guardrails, and evaluation cost.
Answer success per dollar.
Cache hit rate where caching is allowed.
Human escalation and fallback rate.
User correction, abandonment, and re-query rate.

A more accurate configuration can still be unacceptable if it doubles p95 latency, triples cost, or creates an unusable refusal pattern.

Build the Evaluation Contract Before the Dataset

An evaluation contract makes the decision explicit. Without it, teams run metrics first and negotiate what “good” means after seeing the scores.

Document:

Contract field	Example
Decision	Approve retrieval configuration B for controlled production rollout
Population	Internal HR policy questions from employees in India and the UK
Critical failures	Cross-region policy confusion, unauthorized source exposure, invented entitlement
Primary metrics	Exact policy hit@5, faithfulness, correctness, citation coverage
Guardrail metrics	Unauthorized retrieval, harmfulness, sensitive-data leakage
Operational bounds	p95 under 4 seconds; median variable cost below agreed budget
Required reviewers	HR policy owner, security reviewer, product owner
Baseline	Current production configuration A
Candidate	Hybrid search plus reranking configuration B
Release rule	No critical regression; segment gates pass; human review agrees

Avoid one universal threshold copied from another company. A creative research assistant and an eligibility assistant have different error costs. Thresholds should follow business risk, corpus difficulty, user expectations, and escalation options.

Use Severity Before Averages

Classify failures:

Critical: unauthorized data, dangerous instruction, materially false high-impact answer.
High: wrong controlling document, unsupported decision, missing required qualification.
Medium: incomplete but not misleading answer, weak citation coverage, unnecessary refusal.
Low: style, verbosity, minor formatting, non-material wording.

Then make release decisions from both scores and counts. “Correctness improved by 4%” is not a pass if the candidate introduced two critical authorization failures.

Construct a Golden Dataset That Represents Production

The dataset usually matters more than the judge model. A perfectly consistent evaluator cannot compensate for a benchmark containing only simple questions.

Sample Query Types Deliberately

A practical enterprise dataset should cover:

Query stratum	What it exposes
Direct factual lookup	Basic retrieval and answer extraction
Procedural question	Ordered steps and missing prerequisites
Multi-document synthesis	Coverage, conflict resolution, and context limits
Ambiguous terminology	Query clarification and acronym handling
Paraphrase and colloquial language	Semantic robustness
Exact identifier or error code	Keyword and hybrid-search behavior
Table, form, or scanned source	Parsing quality
Time-sensitive question	Version and effective-date control
No-answer question	Abstention and hallucination behavior
Contradictory sources	Source authority and conflict disclosure
Access-restricted question	ACL and tenant isolation
Adversarial source/query	Prompt injection and unsafe behavior
Long multi-turn exchange	Context retention and instruction drift
Rare but high-impact case	Tail-risk protection

Do not let synthetic questions dominate. Start with sanitized production queries, support tickets, search logs, subject-matter expert interviews, and documented incidents. Use synthetic generation to expand phrasing and edge cases, then have domain owners validate the result.

Record More Than an Expected Answer

For every case, store:

{
  "case_id": "hr-india-leave-014",
  "query": "Can unused casual leave be carried into next year?",
  "query_type": "policy_lookup",
  "persona": "employee_india",
  "expected_answer": "No. Casual leave expires at the end of the calendar year.",
  "expected_sources": ["HR-IND-LEAVE-2026#section-4.2"],
  "forbidden_sources": ["HR-UK-LEAVE-2026", "HR-IND-LEAVE-2024"],
  "required_claims": ["casual leave does not carry forward"],
  "required_qualifiers": ["India policy", "calendar year"],
  "answerable": true,
  "risk": "high",
  "owner": "hr-policy-team",
  "as_of": "2026-08-01"
}

The Bedrock prompt-dataset schema may use a subset or transformation of this record. Keep the richer canonical dataset in version control or a governed data catalog, then generate the required JSONL for each job.

Separate Development, Calibration, and Holdout Sets

Development set: visible to engineers; used for rapid iteration.
Calibration set: used to align automated judge results with human scoring and tune thresholds.
Holdout set: not used during tuning; used for release evidence.
Adversarial set: security, abuse, injection, leakage, and access-control cases.
Production shadow set: recent sanitized examples used to detect changing query patterns.

Repeatedly tuning on one benchmark causes evaluation overfitting. The RAG system becomes excellent at the test rather than reliable for the actual population.

Version the Truth

Every case needs source lineage, author, approval status, effective date, and review date. When a policy changes, do not silently overwrite the expected answer. Create a new dataset version and record which system release is evaluated against which knowledge snapshot.

Prepare the Amazon Bedrock Prompt Dataset

Amazon Bedrock expects JSON Lines (.jsonl) in Amazon S3. Each line is one valid JSON object. Current AWS documentation permits up to 1,000 prompts in a RAG evaluation job. Retrieve-only jobs are single-turn; retrieve-and-generate datasets can contain up to five conversation turns.

When Bedrock Invokes a Knowledge Base

For a basic managed retrieve-only or retrieve-and-generate job, each record contains the prompt. Include a reference response when the selected metric requires ground truth or when it helps the evaluator.

{"conversationTurns":[{"prompt":{"content":[{"text":"Can unused casual leave be carried into next year?"}]},"referenceResponses":[{"content":[{"text":"No. Under the 2026 India leave policy, casual leave expires at the end of the calendar year."}]}]}]}

referenceResponses represents the expected end-to-end answer, not the expected raw chunk. AWS specifically notes this distinction for context coverage.

When You Bring Your Own RAG Outputs

To evaluate a custom RAG source, include the prompt, generated answer, retrieved passages, and a knowledgeBaseIdentifier that matches the source name configured for the job.

{"conversationTurns":[{"prompt":{"content":[{"text":"Can unused casual leave be carried into next year?"}]},"referenceResponses":[{"content":[{"text":"No. Under the 2026 India leave policy, casual leave expires at the end of the calendar year."}]}],"referenceContexts":[{"content":[{"text":"Section 4.2: Casual leave expires on December 31 and is not carried forward."}]}],"output":{"text":"Casual leave cannot be carried into the next calendar year.","modelIdentifier":"candidate-generator-v4","knowledgeBaseIdentifier":"custom-rag-b","retrievedPassages":{"retrievalResults":[{"name":"HR-IND-LEAVE-2026#section-4.2","content":{"text":"Section 4.2: Casual leave expires on December 31 and is not carried forward."},"metadata":{"region":"IN","effective_year":"2026"}}]}}}]}

AWS documents referenceContexts as optional for bring-your-own responses and notes that built-in metrics do not use it; it is available for custom metrics. This is another reason to retain your own deterministic expected-source evaluation outside the managed job.

Validate Before Upload

Before storing the file in S3:

Parse every JSONL line independently.
Enforce required fields by evaluation mode.
Reject duplicate case IDs in the canonical dataset.
Verify that the source identifier is consistent across the job.
Scan for secrets and unnecessary personal data.
Confirm that ground truth matches the knowledge snapshot.
Record a content hash for the input file.
Encrypt the bucket and apply a retention policy.

Do not use the production prompt log as an evaluation dataset without data classification and redaction. Evaluation inputs and outputs can contain the same confidential data as the application itself.

Run a Retrieve-Only Evaluation First

The first experiment should answer: “Can this retrieval configuration consistently assemble the evidence required to answer the query?”

Freeze everything that is not under test:

Corpus snapshot.
Parser and chunking configuration.
Embedding model.
Vector index.
Search type.
Metadata filters.
Number of results.
Reranker and candidate count.
Query transformation.
Access persona.

Change one factor at a time when possible. A comparison between “old system” and “new system” where six components changed may identify a winner but cannot explain why it won.

A Retrieval Experiment Matrix

Candidate	Controlled change	Hypothesis
A	Semantic search, K=5	Baseline
B	Hybrid search, K=5	Improve identifier and acronym queries
C	Hybrid search, retrieve 20, rerank to 5	Improve top-rank quality without increasing context
D	Same as C plus policy metadata filters	Reduce obsolete and cross-region sources

Run Bedrock context relevance and coverage for each candidate, then compute deterministic ranking and security metrics from the retrieved IDs. Inspect results by query stratum.

Minimal Deterministic Retrieval Scoring

from statistics import mean

def retrieval_metrics(expected_ids, retrieved_ids, k=5):
    expected = set(expected_ids)
    top_k = retrieved_ids[:k]
    hits = [doc_id for doc_id in top_k if doc_id in expected]

    recall_at_k = len(set(hits)) / len(expected) if expected else 1.0
    precision_at_k = len(hits) / k if k else 0.0
    first_rank = next(
        (rank for rank, doc_id in enumerate(top_k, start=1) if doc_id in expected),
        None,
    )
    reciprocal_rank = 1 / first_rank if first_rank else 0.0

    return {
        "recall_at_k": recall_at_k,
        "precision_at_k": precision_at_k,
        "reciprocal_rank": reciprocal_rank,
    }

cases = [
    retrieval_metrics(["policy-2026#4.2"], ["faq-7", "policy-2026#4.2"], 5),
    retrieval_metrics(["benefits-2026#8"], ["benefits-2024#8"], 5),
]

summary = {
    key: mean(case[key] for case in cases)
    for key in cases[0]
}

Production code should also calculate confidence intervals, segment results, preserve the ranked lists, and flag forbidden sources.

Diagnose Retrieval Failures Before Tuning the Model

Observed failure	Likely investigation
Expected source never appears	Ingestion, parser, embedding, filter, or index issue
Expected source appears below K	Search mode, ranking, reranker, chunk representation
Correct document but wrong passage	Chunk boundaries, table parsing, parent-child retrieval
Many relevant but repetitive chunks	Deduplication, parent grouping, diversity selection
Expired document ranks first	Metadata, source authority, effective-date logic
Results cross a security boundary	Identity propagation, ACL filters, tenant partitioning
Multi-part question has partial evidence	Query decomposition, K, multi-hop retrieval

Do not compensate for a retrieval defect with a more capable generator. A model may infer the right answer during testing, but the system remains unsupported and fragile.

Run Retrieve-and-Generate Evaluation

Once retrieval has a credible baseline, evaluate the response layer. Keep the retrieval configuration fixed while comparing generator models, prompts, context formatting, answer policies, or citation behavior.

Create a Managed Evaluation Job

In the Amazon Bedrock console, the current workflow is under Inference and assessment → Evaluations → RAG evaluations. Select an evaluator model, the inference source, the evaluation type, metrics, input and output S3 locations, and an IAM service role. You may use a customer-managed AWS KMS key; otherwise AWS documents use of an AWS-owned key for the job data.

The same capability is available through CreateEvaluationJob. The following abbreviated CLI configuration reflects the current AWS API shape for a Bedrock Knowledge Base; replace identifiers and confirm supported models in the deployment Region:

{
  "jobName": "hr-rag-release-2026-08",
  "jobDescription": "Evaluate candidate B on holdout dataset v12",
  "roleArn": "arn:aws:iam::123456789012:role/bedrock-rag-eval-role",
  "applicationType": "RagEvaluation",
  "evaluationConfig": {
    "automated": {
      "datasetMetricConfigs": [
        {
          "taskType": "General",
          "dataset": {
            "name": "hr-holdout-v12",
            "datasetLocation": {
              "s3Uri": "s3://company-ai-evals/input/hr-holdout-v12.jsonl"
            }
          },
          "metricNames": [
            "Builtin.Correctness",
            "Builtin.Completeness",
            "Builtin.Faithfulness",
            "Builtin.CitationPrecision",
            "Builtin.CitationCoverage",
            "Builtin.Refusal"
          ]
        }
      ],
      "evaluatorModelConfig": {
        "bedrockEvaluatorModels": [
          {"modelIdentifier": "SUPPORTED_EVALUATOR_MODEL_ID_OR_PROFILE"}
        ]
      }
    }
  },
  "inferenceConfig": {
    "ragConfigs": [
      {
        "knowledgeBaseConfig": {
          "retrieveAndGenerateConfig": {
            "type": "KNOWLEDGE_BASE",
            "knowledgeBaseConfiguration": {
              "knowledgeBaseId": "KNOWLEDGE_BASE_ID",
              "modelArn": "SUPPORTED_GENERATOR_MODEL_ARN"
            }
          }
        }
      }
    ]
  },
  "outputDataConfig": {
    "s3Uri": "s3://company-ai-evals/output/hr-rag-release-2026-08/"
  }
}

Run it with:

aws bedrock create-evaluation-job --cli-input-json file://rag-eval-job.json

The API returns an evaluation-job ARN and processes asynchronously. Use GetEvaluationJob or the console to inspect status, and store the job ARN in the experiment record.

Use Least-Privilege Evaluation Roles

AWS requires a service role that Bedrock can assume. Scope it to:

The exact input and output S3 prefixes.
The selected evaluator model.
The selected response-generator model when Bedrock generates responses.
bedrock:Retrieve and/or bedrock:RetrieveAndGenerate for the target Knowledge Base.
The specific KMS key when customer-managed encryption is used.

Separate the human or CI role that creates jobs from the service role assumed by Bedrock. Add source-account and evaluation-job source-ARN conditions to the trust policy following the AWS service-role guidance.

Preserve the Complete Experiment Manifest

experiment_id: hr-rag-2026-08-18-b
decision: release-candidate
dataset:
  name: hr-holdout
  version: 12
  sha256: "..."
knowledge_snapshot: 2026-08-15T18:00:00Z
retrieval:
  search_type: hybrid
  initial_k: 20
  final_k: 5
  reranker: "..."
generation:
  model: "..."
  prompt_version: answer-policy-v9
evaluation:
  bedrock_job_arn: "..."
  evaluator_model: "..."
  custom_metric_versions:
    - policy-version-use-v3
owners:
  engineering: rag-platform
  business: hr-policy

Without this manifest, a higher score may be impossible to reproduce three weeks later.

Design Custom Metrics That Reflect the Business

A custom LLM judge is useful when quality depends on domain meaning rather than exact text. The prompt should define a role, task, criterion, observable scoring rules, and input variables. AWS recommends placing input variables last.

Example: evaluate whether an answer uses the controlling source rather than an obsolete policy.

{
  "customMetricDefinition": {
    "metricName": "controlling_policy_use",
    "instructions": "You are reviewing an enterprise policy answer. Score whether the answer follows the currently controlling policy in the supplied context, explicitly handles conflicts with older policy text, and does not invent precedence. Score 0 when it follows an obsolete or conflicting source, 1 when the controlling source is unclear or the answer omits a necessary qualification, and 2 when it follows the controlling source and accurately explains any material conflict. Evaluate only the supplied inputs.\n\nQuestion: {{prompt}}\nRetrieved context: {{context}}\nResponse: {{prediction}}",
    "ratingScale": [
      {"definition": "Wrong policy", "value": {"floatValue": 0}},
      {"definition": "Unclear or incomplete", "value": {"floatValue": 1}},
      {"definition": "Correct controlling policy", "value": {"floatValue": 2}}
    ]
  }
}

Custom-Metric Design Rules

Score one coherent property per metric.
Define observable distinctions between scale points.
Include examples in the rubric when ambiguity is likely.
Do not ask the judge to validate facts it cannot see.
Avoid brand, style, and policy criteria in one combined score.
Include a path for “insufficient information” where appropriate.
Test order sensitivity by changing response order where comparisons are involved.
Version the prompt and scale.
Preserve the metric definition produced in the output S3 location.
Calibrate against domain experts before using it as a release gate.

Custom metrics are still model judgments. They are not deterministic policy engines, legal review, or evidence that a requirement is satisfied in every case.

Calibrate the LLM Judge With Human Review

An evaluator model can be consistent, scalable, and useful while still disagreeing with the people responsible for the business outcome.

Potential judge errors include:

Preferring verbosity over concise correctness.
Rewarding lexical overlap with a reference response.
Missing a subtle domain exception.
Accepting a plausible citation that does not support the precise claim.
Penalizing a necessary refusal.
Scoring its own model family's writing style more favorably.
Changing behavior after model updates.

A Practical Calibration Process

Select a stratified sample containing clear passes, clear failures, and borderline cases.
Have at least two qualified reviewers score independently with the same rubric.
Resolve disagreements and refine the rubric.
Run the Bedrock evaluator on the same cases.
Measure agreement, rank correlation, false passes, and false failures.
Investigate systematic disagreement by query type and severity.
Adjust the metric prompt or release threshold.
Lock the evaluator model and metric version for the comparison.

Do not evaluate calibration only with Pearson correlation on aggregate scores. For high-risk applications, the false-pass rate on critical cases matters more. A judge that agrees 95% overall but approves two dangerous answers is not fit for the release gate.

Human review remains especially important for:

High-impact policy or compliance answers.
Ambiguous and multi-document reasoning.
New languages or jurisdictions.
Novel failure types.
Sampled production traffic.
Final go/no-go approval.

The best operating model combines deterministic checks, Bedrock LLM judging, and targeted expert review.

Turn Scores Into Enterprise Release Gates

Scores become useful only when they change a decision.

The following is an illustrative gate not a universal benchmark:

Dimension	Illustrative gate	Rule
Exact authorized source hit@5	≥ 0.95	Must pass for every high-risk segment
Recall@5	≥ 0.90	No segment may regress more than 0.02
Bedrock context relevance	≥ 0.80	Inspect histogram and bottom decile
Correctness	≥ 0.88	Zero critical false answers in holdout
Faithfulness	≥ 0.92	Human-confirm bottom-decile cases
Citation precision	≥ 0.90	Pair with citation coverage
Citation coverage	≥ 0.90	No uncited high-impact conclusion
Harmfulness	≤ 0.02	Lower is better
Unauthorized retrieval	0	Hard fail
p95 latency	≤ 4 seconds	Measured under target concurrency
Cost per successful answer	Within budget	Include all RAG components

Do Not Hide Segment Failures in an Average

Report at least:

Overall mean and median.
Distribution or histogram.
Bottom decile.
Pass rate at the case threshold.
Critical and high-severity failure counts.
Results by query type, persona, language, source, risk, and answerability.
Confidence intervals for candidate-versus-baseline differences.

Suppose candidate B improves overall correctness from 0.84 to 0.88 but reduces correctness for time-sensitive policy questions from 0.91 to 0.73. The overall average recommends B; the business risk rejects it.

Use Paired Comparisons

Evaluate baseline and candidate on the same cases. Report per-case deltas and use paired statistical methods or bootstrap confidence intervals. This removes some dataset variance and makes the changed behavior inspectable.

Do not claim an improvement from a difference smaller than judge variability. Rerun a sample, evaluate with a second judge where warranted, and confirm with humans.

Worked Example: Choosing Between Two Bedrock RAG Configurations

Consider an illustrative multinational manufacturer evaluating a technical-service assistant. The corpus contains repair manuals, service bulletins, product variants, and superseded safety notices.

Candidate A uses semantic retrieval with five results. Candidate B uses hybrid retrieval, retrieves 20 candidates, applies reranking, then sends five passages to the generator. The generator model and prompt are fixed.

The team creates 420 cases:

140 direct part and error-code lookups.
90 troubleshooting procedures.
60 product-variant questions.
50 multi-document questions.
30 superseded-document conflicts.
25 unanswerable questions.
25 authorization and adversarial cases.

Retrieve-only evaluation shows B has higher context relevance and coverage. Deterministic analysis reveals the main improvement: exact identifiers and error codes are more likely to appear in the first three results. However, B also retrieves an obsolete safety bulletin in 11 cases because the reranker favors semantic similarity over effective date.

The team does not proceed directly to the generator comparison. It adds a metadata rule for product family, document status, and effective date, producing Candidate B2. B2 preserves the recall gain and eliminates obsolete-document violations in the holdout set.

Retrieve-and-generate evaluation then shows:

Higher correctness and completeness for troubleshooting procedures.
Similar faithfulness overall.
Better citation coverage.
A higher refusal score for product-variant questions.

Failure inspection shows that the prompt requires refusal whenever a serial number is missing even when the retrieved manual contains a procedure shared by every variant. The team changes the response policy to ask for a serial number only when the answer actually depends on the variant.

The final release is not “hybrid search won.” The evidence is more precise:

Hybrid retrieval plus reranking improved identifier recall, metadata authority rules prevented obsolete evidence, and a narrower clarification policy reduced unnecessary refusals without weakening safety.

That statement is actionable, repeatable, and attributable to controlled changes.

Use a Failure Taxonomy, Not a Screenshot Folder

For every failed case, assign the earliest responsible stage and a specific subtype.

Stage	Failure subtype	Example remediation
Corpus	Missing, stale, duplicated, unapproved	Fix source governance and ingestion
Parsing	Lost table, OCR error, heading detached	Change parser or document preparation
Chunking	Split rule, context fragmentation, oversized chunk	Tune chunk strategy or hierarchical retrieval
Retrieval	Miss, low rank, wrong search mode	Hybrid search, query rewrite, embeddings
Filtering	Over-filter, under-filter, ACL defect	Fix metadata and identity propagation
Reranking	Correct evidence demoted	Tune candidate pool or reranker
Context assembly	Relevant evidence dropped or repeated	Budgeting, deduplication, ordering
Generation	Unsupported claim, omission, contradiction	Prompt, model, response policy
Citation	Wrong span, missing source, broken mapping	Citation construction and validation
Safety	Harm, injection, leakage	Guardrails plus application controls
Product	Poor clarification, unusable format	UX and interaction policy
Operations	Latency, throttle, timeout, excessive cost	Capacity, caching, fallback, limits

rack failure counts over time. If 38% of failures originate in parsing, changing the evaluator or generator model is noise. If most remaining failures are “correct evidence retrieved but unsupported claim generated,” the generator and prompt deserve attention.

This failure-first approach also supports a focused audit of an underperforming RAG system instead of a costly rebuild based on intuition.

Evaluate Security and Guardrails Separately

RAG quality includes safe failure, but Bedrock RAG evaluation metrics are not a complete security test.

Create adversarial suites for:

Direct jailbreaks and prompt injection.
Instructions hidden in retrieved documents.
Attempts to extract system prompts or secrets.
Cross-user and cross-tenant document requests.
PII, credentials, and confidential identifiers.
Encoded or multilingual attack variants.
Malicious file content.
Tool-use escalation if the RAG assistant can take actions.

Amazon Bedrock Guardrails can add content filters, prompt-attack detection, denied topics, sensitive-information handling, contextual grounding, and other policies depending on configuration. It does not replace retrieval authorization, tenant isolation, source trust, or deterministic tool controls. Our separate guide explains how to secure enterprise AI with Amazon Bedrock Guardrails.

For each adversarial case, record:

Whether unauthorized evidence was retrieved.
Whether unsafe text reached the generator.
Whether the model followed the malicious instruction.
Whether the response exposed protected information.
Whether the guardrail intervened.
Whether the application failed closed or exposed a partial response.
Whether the event produced the correct security telemetry.

A response can be faithful to malicious retrieved text. Faithfulness is therefore not equivalent to safety.

Evaluate Latency and Cost With Quality

Optimization is multi-objective. Evaluate the Pareto frontier rather than maximizing a single metric.

Measure the Full Request Path

Break latency and cost into:

Query classification or transformation.
Embedding or lexical-query generation.
Vector, keyword, graph, or structured retrieval.
Reranking.
Context assembly.
Guardrail input processing.
Generation.
Guardrail output processing.
Citation validation and post-processing.

For evaluation itself, AWS charges the evaluator-model token usage at the model's on-demand standard-tier rates. AWS also states that evaluating a Bedrock Knowledge Base incurs its normal Knowledge Base usage charges. Generator inference and any optional components still contribute. Check the live Amazon Bedrock pricing page rather than embedding a static total.

Calculate Cost per Successful Answer

Cost per request can reward a cheap configuration that frequently fails. A better economic measure is:

cost_per_successful_answer = total_evaluated_system_cost / number_of_cases_passing_all_required_gates

Also estimate the cost of false answers, human escalation, support remediation, and compliance review. The cheapest token path may have the highest business cost.

Move Evaluation Into CI/CD Without Running Everything on Every Commit

Use evaluation tiers:

Tier	Trigger	Dataset	Purpose
Smoke	Every relevant code change	20–50 deterministic cases	Catch obvious schema, retrieval, and prompt breaks
Regression	Merge or daily	Representative development set	Catch component regressions
Release	Production candidate	Holdout plus adversarial sets	Formal go/no-go evidence
Scheduled	Weekly or monthly	Full benchmark and recent shadow data	Detect corpus and judge drift
Incident	Quality or security alert	Related cases plus new reproductions	Confirm remediation and prevent recurrence

The pipeline should fail immediately on deterministic hard gates such as unauthorized retrieval. Bedrock evaluation jobs are asynchronous, so the CI system can create a job, store the ARN, poll with a bounded timeout, download the S3 results, calculate segment gates, and publish an evaluation artifact.

Version or fingerprint:

Application code.
Knowledge corpus snapshot.
Parser and chunking configuration.
Embedding model.
Index schema.
Retrieval and reranking settings.
Generator model and inference parameters.
Prompt templates.
Guardrail versions.
Dataset.
Evaluator model.
Built-in metric list.
Custom metric definitions.

If any of these changes, the evaluation result represents a new system candidate.

Monitor RAG Quality After Release

Offline evaluation answers whether a fixed candidate performs well on a fixed dataset. Production changes continuously:

Documents are added, removed, and revised.
User vocabulary changes.
Query distribution shifts.
Permissions change.
Providers update models.
Indexes are rebuilt.
Latency and throttling vary with load.
New attack patterns appear.

Build an online evaluation loop:

Capture trace IDs, system versions, retrieval IDs, applied filters, response, citations, latency, token usage, and policy decisions.
Minimize and protect logged content according to its classification.
Sample traffic by risk and query type, not only at random.
Automatically score eligible samples with the same versioned rubrics.
Route low-confidence and high-risk cases to human review.
Convert confirmed failures into regression cases.
Compare production distributions with the benchmark.
Alert on sustained changes rather than isolated noisy judge scores.

Production Signals That Deserve Investigation

Rising re-query or reformulation rate.
More “not helpful” feedback in one query segment.
Higher refusal rate after a Guardrail or prompt change.
Declining citation click-through where citations are part of the workflow.
Increasing retrieval from obsolete sources.
Increased empty retrievals.
Model answers that lack cited support.
Higher human escalation rate.
Latency or cost growth without quality gain.
Sudden judge-score changes after evaluator-model updates.

Do not send raw confidential queries to an evaluation path without confirming data handling, access, encryption, retention, and regional requirements.

Common Evaluation Mistakes

Testing Only the Final Answer

This hides whether the correct answer came from retrieval, model memory, or luck. Preserve and score the evidence.

Using Only Built-In Metrics

Built-in metrics provide a strong baseline but do not encode business-specific authority, format, escalation, or security requirements. Add deterministic and custom checks.

Treating Every High Score as Good

Harmfulness, stereotyping, and refusal measure the presence of those characteristics. Higher can be worse. Normalize polarity before creating an executive scorecard.

Choosing Thresholds After Seeing the Result

This turns the test into a negotiation. Define the evaluation contract and release rule first.

Comparing Uncontrolled Configurations

If corpus, retrieval, prompt, and model change together, the team cannot attribute improvement or regression.

Trusting Averages

Averages conceal failures for specific languages, products, permissions, and high-risk tasks. Segment the results and inspect the tails.

Using LLM Judges as Ground Truth

The judge is another model. Calibrate it, version it, and combine it with deterministic checks and people.

Ignoring Negative and Unanswerable Cases

An assistant must know when evidence is absent, outdated, unauthorized, or ambiguous. False confidence is often more damaging than refusal.

Reusing Production Data Without Governance

Queries and results can contain personal, confidential, or regulated information. Sanitize and control evaluation data as production data.

Running an Evaluation Once

A static report decays as the system changes. Evaluation must become a release and monitoring capability.

When Amazon Bedrock RAG Evaluation Is the Right Fit

It is especially useful when:

The application already uses Amazon Bedrock Knowledge Bases.
The team wants managed LLM-as-a-judge scoring within AWS.
Several Bedrock Knowledge Base configurations must be compared.
A custom or external RAG system can export precomputed passages and responses.
Built-in semantic metrics cover much of the baseline need.
The organization wants S3-based results, IAM roles, and optional customer-managed KMS encryption.
Evaluation jobs need to be launched through the console, CLI, or SDK.

It is not sufficient by itself when:

Exact document identity and ranking determine correctness.
Authorization defects must be proved absent.
The use case requires extensive human preference research.
Online monitoring and real-time quality controls are required.
Tool execution or multi-agent trajectories must be evaluated.
The application needs metrics unsupported by the managed input/output format.
The organization requires evaluator models or Regions not currently supported.
High-risk decisions require formal validation rather than probabilistic judging.

The sensible architecture is often Bedrock managed evaluation plus a customer-owned evaluation harness.

Enterprise RAG Evaluation Checklist

Scope and Governance
[ ] The business decision and evaluation owner are documented.
[ ] Critical, high, medium, and low failure classes are defined.
[ ] Release thresholds were approved before the final run.
[ ] Dataset, metric, evaluator, and system versions are preserved.
[ ] Data-classification, retention, and regional requirements are approved.

Dataset
[ ] Queries represent production strata and languages.
[ ] Authoritative reference answers have named owners.
[ ] Expected and forbidden sources are captured.
[ ] Answerable, unanswerable, ambiguous, stale, and conflicting cases exist.
[ ] ACL and adversarial cases exist.
[ ] Development, calibration, and holdout sets are separated.
[ ] The JSONL file validates and its hash is recorded.

Retrieval
[ ] Retrieval is evaluated independently from generation.
[ ] Recall@K, MRR/nDCG, and exact-source metrics are calculated where applicable.
[ ] Bedrock context relevance and coverage are interpreted correctly.
[ ] Results are segmented by query and document type.
[ ] Unauthorized retrieval and obsolete-source violations are hard gates.

Generation and Citations
[ ] Correctness, completeness, and faithfulness are evaluated.
[ ] Citation precision and coverage are used together.
[ ] Refusal is separated for answerable and unanswerable queries.
[ ] Harmfulness and stereotyping are treated as lower-is-better metrics.
[ ] Domain requirements use versioned custom metrics or deterministic rules.

Calibration and Operations
[ ] Human reviewers calibrated the evaluator on representative cases.
[ ] False passes on high-severity cases are measured.
[ ] Latency, error rate, and cost are evaluated with quality.
[ ] CI/CD has smoke, regression, and release tiers.
[ ] Production sampling feeds confirmed failures back into the benchmark.
[ ] Rollback and incident procedures are tested.

Frequently Asked Questions

What is Amazon Bedrock RAG evaluation?

It is a managed evaluation capability that uses supported evaluator models to score how a Bedrock Knowledge Base or another RAG source retrieves information and, optionally, generates responses. Jobs consume a JSONL dataset from S3 and produce reports and result artifacts.

Can Amazon Bedrock evaluate a custom RAG system?

Yes. AWS supports bring-your-own inference response data. Supply the query, retrieved passages, generated response where applicable, and a source identifier in the documented JSONL structure. Bedrock then skips invoking a Knowledge Base and evaluates the supplied output.

What is the difference between retrieve-only and retrieve-and-generate evaluation?

Retrieve-only evaluates retrieved context, making it appropriate for chunking, search, filter, and reranking experiments. Retrieve-and-generate evaluates the final response and citations as well as the supplied retrieval context, making it appropriate for model and prompt comparisons.

Which retrieval metrics does Bedrock provide?

The current built-in retrieve-only metrics are context relevance and context coverage. Coverage requires a ground-truth reference response. Add deterministic Recall@K, Precision@K, MRR, nDCG, source-version, and authorization checks when exact retrieval behavior matters.

Which answer-quality metrics does Bedrock provide?

Current built-in metrics include correctness, completeness, helpfulness, logical coherence, faithfulness, citation precision, citation coverage, harmfulness, stereotyping, and refusal.

Are higher Bedrock RAG evaluation scores always better?

No. A score closer to 1 means more of the named characteristic is present. Higher correctness and faithfulness are favorable; higher harmfulness and stereotyping are unfavorable. Refusal must be interpreted against whether a query should be answered.

Does faithfulness mean the answer is correct?

No. Faithfulness asks whether the answer is supported by retrieved evidence. If the evidence is obsolete, malicious, or wrong, a faithful answer can still be factually or operationally incorrect. Measure source authority and correctness separately.

How large should a RAG evaluation dataset be?

There is no universal number. Coverage across important query strata matters more than raw size. Current Bedrock documentation permits up to 1,000 prompts per RAG evaluation job. Enterprises can maintain a larger canonical benchmark and create versioned job subsets.

Can Bedrock evaluate multi-turn RAG conversations?

Current AWS documentation allows up to five conversation turns for retrieve-and-generate evaluation datasets. Retrieve-only evaluation is single-turn. Test longer product conversations through a customer-owned harness if needed.

Should we use an LLM as a judge?

Use it for scalable semantic assessment, but calibrate it against domain experts. Keep deterministic checks for exact facts, source IDs, permissions, schemas, and business rules. Use human review for critical and ambiguous cases.

How do we compare two Knowledge Bases fairly?

Use the same dataset, knowledge snapshot, evaluator model, metric versions, and release thresholds. Change one architectural factor at a time where possible. Compare paired case results and segments, not only overall means.

How much does Bedrock RAG evaluation cost?

AWS charges evaluator-model token usage at the selected model's on-demand standard-tier pricing. A RAG evaluation that invokes a Bedrock Knowledge Base also incurs its usual usage charges, plus applicable generator and optional component costs. Calculate the full experiment before large runs and verify current pricing.

Does Bedrock RAG evaluation test document permissions?

Not as a deterministic authorization proof. Create persona-based positive and negative retrieval tests and make any unauthorized source or citation a hard failure. Security must be evaluated independently from semantic answer quality.

Can evaluation run in CI/CD?

Yes. Create jobs through the AWS CLI or SDK, poll the asynchronous job, process S3 results, calculate customer-owned metrics, and enforce release gates. Use small smoke tests frequently and larger Bedrock judge runs at merge, release, or scheduled intervals.

How often should a production RAG system be reevaluated?

Evaluate after material changes to corpus, parsing, chunking, embeddings, retrieval, reranking, models, prompts, Guardrails, or authorization. Also run scheduled tests and sample production traffic so query and corpus drift become visible.

What This Means for Your Organization

The goal is not to produce an attractive scorecard. It is to create evidence that supports a release, explains failures, guides engineering work, and detects regressions after deployment.

A credible enterprise RAG evaluation program should be able to answer:

What population does this benchmark represent?
Which evidence should each query retrieve?
Which failures are unacceptable even once?
Which system change caused the score to move?
Does the LLM judge agree with qualified reviewers?
Can the team reproduce the result from recorded versions?
What happens when the corpus or query distribution changes?

If those questions cannot be answered, the organization has a demo score—not an evaluation system.

How Codersarts Helps Evaluate and Improve Enterprise RAG

Codersarts can add an evaluation layer to an existing Amazon Bedrock RAG system or design the evaluation program alongside a new implementation.

Our work can include:

Failure-mode and risk discovery.
Golden-dataset construction with domain owners.
Deterministic retrieval and authorization metrics.
Amazon Bedrock RAG evaluation job setup.
Custom LLM-as-a-judge rubric design and calibration.
Human-review workflows.
Retrieval, chunking, reranking, prompt, and citation experiments.
CI/CD regression gates.
Production sampling and evaluation dashboards.
Security and adversarial testing.
Root-cause analysis and remediation planning.

Explore our LLM Evaluation and Benchmark Engineering service or RAG Development Services. For the broader implementation approach, read How We Measure RAG Accuracy and How to Build Enterprise RAG with Amazon Bedrock Knowledge Bases.

Need a measurable answer to whether your RAG system is ready to ship?

Discuss your RAG evaluation or remediation requirement with Codersarts. Bring the current architecture, a sample corpus, representative queries, and known failures. We can turn them into a reproducible baseline and a prioritized improvement plan.

Official Amazon Bedrock References

Editorial note: Amazon Bedrock evaluation features, evaluator and generator models, Regions, quotas, schemas, metric behavior, and pricing change. Verify the official documentation in the target Region before publishing an implementation commitment or running a production evaluation.