Production Observability for AI Agents on AWS: Traces, Latency, Tokens and Failures

A conventional API can be healthy when it returns a successful status code within its latency objective. An AI agent can return 200 OK and still fail its user. It may choose the wrong tool, pass a valid but dangerous parameter, retrieve outdated evidence, loop through unnecessary model calls, consume ten times the normal tokens, or produce a fluent answer that does not complete the task.

That changes the meaning of production observability.

For an AI agent, infrastructure health is necessary but incomplete. Operations teams must see the execution path, model and tool dependencies, token consumption, policy decisions, state behavior, output quality, and business outcome without turning prompts, credentials, and private records into a second ungoverned data store.

This guide presents an AWS-native observability design for agents running on Amazon Bedrock AgentCore, Lambda, ECS, EKS, or EC2. It uses Amazon CloudWatch generative AI observability, CloudWatch Transaction Search, AWS Distro for OpenTelemetry (ADOT), Amazon Bedrock runtime metrics and invocation logs, AgentCore Evaluations, and application-defined business signals. The goal is not more telemetry. The goal is faster, safer decisions when the agent behaves differently than expected.

Direct answer: instrument the complete agent task as one distributed trace; create child spans for orchestration, model, retrieval, policy, memory, and tool work; record bounded, non-sensitive attributes; combine AgentCore service metrics with Bedrock token and invocation metrics; emit a separate business-success signal; classify failures by layer; and alert on SLO burn, critical safety events, token anomalies, and failure clusters. Preserve full diagnostic traces selectively, not every prompt by default.

The Observability Questions a Production Team Must Answer

An observability platform is useful only if it answers operational questions. For an agent, the minimum set is broader than “is the endpoint up?”

Question	Signal required	Primary AWS source
Did the request reach the runtime?	invocation count, HTTP/API outcome	AgentCore Runtime or hosting-service metrics
Did the agent complete the user's task?	business outcome and evaluator result	application metric and AgentCore Evaluations
Why was the response slow?	end-to-end trace and span latency	AgentCore Observability, ADOT, CloudWatch Transaction Search
How many model calls and tokens were used?	model spans, Bedrock usage fields, invocation logs	Amazon Bedrock and application telemetry
Which tool was selected and with what validated parameters?	tool spans and audit events	agent instrumentation, Gateway, downstream system
Was access allowed for the right reason?	identity and policy decision	AgentCore Identity/Gateway Policy telemetry and CloudTrail
Did retrieval return authorized, current evidence?	retrieval spans, source identifiers, freshness	application/RAG telemetry
Did the agent loop, retry, or fall back?	graph transitions, attempts, termination reason	framework and custom spans
Is one tenant, model, version, or intent failing disproportionately?	low-cardinality dimensions and segmented evaluation	metrics, logs, traces, evaluation results
Is telemetry itself missing or delayed?	heartbeat and telemetry-delivery health	CloudWatch delivery metrics and synthetic canaries

If the team cannot answer these questions for a specific failed request, it has monitoring, not observability.

A Practical Telemetry Model: Six Layers, One Trace

The cleanest architecture gives every user-visible task a trace and represents each major dependency as a child span.

User or calling service
        |
        v
API / identity / rate limit
        |
        v
Agent runtime session ------------------ runtime metrics
        |
        +──▶ orchestration / graph ────── node and transition spans
        |        |
        |        +──▶ model call ──────── latency, tokens, model errors
        |        +──▶ retrieval ───────── filters, source IDs, freshness
        |        +──▶ policy ──────────── allow/deny and reason category
        |        +──▶ tool call ───────── validated operation and outcome
        |        +──▶ memory ──────────── read/write, hit, age, actor scope
        |
        v
Response and business outcome ---------- success, escalation, abandonment

Telemetry destinations:
CloudWatch metrics + structured logs + Transaction Search + evaluations

Layer 1: Entry and identity

Capture the API operation, environment, agent endpoint, deployment version, authentication mode, request class, and a pseudonymous caller or tenant reference. Do not place a raw access token, authorization header, email address, or customer name in trace baggage.

Layer 2: Runtime and session

Capture Runtime invocation latency, new and active session counts, throttles, user errors, system errors, CPU, memory, and streaming connections where applicable. A stable session ID connects multiple request traces into one conversation, but it must not double as authorization proof.

Layer 3: Orchestration

Record graph nodes, decisions, retries, interrupts, fallbacks, termination reason, step count, and state-store operations. The trace should reveal that the agent looped from plan to tool four times; a single “agent latency = 19 seconds” metric cannot.

Layer 4: Models and retrieval

Capture model or inference-profile identifier, call latency, time to first token for streaming, input/output/cache token counts where returned, stop reason, retrieval latency, result count, evidence identifiers, and freshness. Prompt or retrieved text should be off by default unless a reviewed logging mode permits it.

Layer 5: Tools, policy, and side effects

Record the logical tool name, schema version, validation outcome, authorization decision, downstream status, retry count, idempotency reference, and receipt. Do not record secrets or unrestricted tool payloads.

Layer 6: User and business outcome

Emit whether the task completed, required escalation, was corrected by the user, was abandoned, or caused downstream rework. A technically successful trace is not a successful agent run until the intended outcome is verified.

Understand the CloudWatch Signal Sources

AWS exposes related telemetry through different namespaces and storage paths. Treat them as complementary, not interchangeable.

Source	Typical namespace or location	Best for	Important limitation
AgentCore service metrics	AWS/Bedrock-AgentCore	Runtime, Gateway, Memory, Identity, Policy, and built-in service health	does not know whether the user's business task succeeded
Instrumented agent metrics	bedrock-agentcore via EMF	framework, graph, custom latency, outcome, and domain signals	schema and cardinality are your responsibility
Bedrock model runtime metrics	AWS/Bedrock	model invocations, latency, input/output tokens, throttles, errors, TTFT	aggregated by supported dimensions, not a full agent trajectory
Bedrock Guardrails metrics	AWS/Bedrock/Guardrails	interventions, text units, latency, errors, policy dimensions	intervention is not automatically a defect or a successful outcome
AgentCore/runtime logs and spans	Runtime log group or aws/spans	request-level diagnosis and trace waterfalls	sampled/retained data may not represent all traffic
Bedrock model invocation logs	configured CloudWatch Logs and/or S3 destination	per-invocation tokens, model ID, identity, optional request metadata and content	disabled by default; content logging creates privacy and cost obligations
CloudTrail	trails or CloudTrail Lake	who changed or invoked supported AWS resources and APIs	audit plane, not detailed application performance telemetry

This distinction prevents three common mistakes:

1. estimating exact billing from a sampled agent dashboard;

   
   

2. treating AWS/Bedrock model latency as the user's complete task latency; and

   
   

3. treating a successful Runtime invocation as proof of a successful business outcome.

Trace the Whole Task, Not Only the Model Call

A production trace should begin when the application accepts the task and end when it returns or durably records an outcome. The model is one span inside it.

Example trace:

agent.task                                            8.42 s
├── identity.resolve                                  0.06 s
├── memory.load                                       0.18 s
├── graph.classify                                    0.03 s
├── gen_ai.chat model=approved-profile                1.91 s
├── retrieval.search                                  0.62 s
│   └── opensearch.query                              0.51 s
├── gen_ai.chat model=approved-profile                3.76 s
├── tool.create_ticket                                1.31 s
│   ├── policy.evaluate                               0.04 s
│   └── ticketing.post                                1.18 s
├── output.validate                                   0.08 s
└── outcome.record                                    0.02 s

Propagate standard trace context

AgentCore supports standard trace propagation headers, including the AWS X-Ray X-Amzn-Trace-Id format and W3C traceparent. Use one consistently across the API layer, Runtime, Gateway, tools, queues, and downstream services. Preserve tracestate only when required by the tracing design.

For AgentCore Runtime HTTP sessions, propagate X-Amzn-Bedrock-AgentCore-Runtime-Session-Id. The session ID groups related interactions and helps route them consistently, while the trace ID identifies one execution. They are not the same identifier.

Use OpenTelemetry baggage sparingly. Baggage propagates, so high-cardinality or sensitive values can spread into systems that were never approved to store them.

Adopt stable span names and attributes

Choose low-cardinality span names:

agent.task
graph.classify
graph.plan
gen_ai.chat
retrieval.search
policy.evaluate
tool.get_order
memory.retrieve
output.validate
outcome.record

Put variable values in attributes, not names. Use tool.get_order with tool.operation=get_order; do not create a span named tool.get_order.order_718293.customer_49281.

Recommended attributes:

service.name
deployment.environment
service.version
agent.name
agent.version
agent.framework
agent.request.class
agent.outcome
agent.termination.reason
agent.step.count
gen_ai.system
gen_ai.request.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
tool.name
tool.schema.version
tool.outcome
policy.decision
error.type

Follow current OpenTelemetry generative AI semantic conventions where available, but version your internal attribute contract. Semantic conventions and framework instrumentors evolve; dashboards must not silently break when an attribute is renamed.

Instrument a LangGraph Agent with ADOT and OpenTelemetry

AgentCore provides service metrics by default. Detailed framework spans and custom metrics require agent instrumentation. For a Python LangGraph application, add:

aws-opentelemetry-distro>=0.10.0
opentelemetry-instrumentation-langchain
langgraph
langchain-aws
bedrock-agentcore

Pin the exact tested versions in the project lock file. Do not copy a floating lower-bound dependency list directly into a production build.

Configure AgentCore-hosted telemetry

Enable CloudWatch Transaction Search once for the account and Region, including span ingestion as structured logs. AgentCore can send spans to the Runtime log group:

/aws/bedrock-agentcore/runtimes/<agent_id>-<endpoint_name>

or to the shared aws/spans destination, depending on configuration. The Runtime log group can contain:

• standard application output in Runtime log streams;

  
  

• structured OpenTelemetry logs;

  
  

• a spans stream when configured as the span destination; and

  
  

• optional application and resource-usage logs configured for the resource.

  
  
  
  

For agents hosted outside AgentCore Runtime, AWS documents ADOT SDK or the AWS Lambda Layer for OpenTelemetry as the supported path for AgentCore observability. The ADOT Collector is not the supported setup for this particular agent-observability integration.

Add manual business spans and metrics

Auto-instrumentation captures framework activity, but it does not know what “successful” means for your organization. Add manual spans and bounded metrics around the task:

import hashlib
import time
from typing import Any, Callable

from opentelemetry import metrics, trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("com.codersarts.support-agent")
meter = metrics.get_meter("com.codersarts.support-agent")

task_counter = meter.create_counter(
    "agent.task.count",
    description="Count of agent tasks by bounded outcome",
)
task_latency = meter.create_histogram(
    "agent.task.duration",
    unit="s",
    description="End-to-end agent task duration",
)
task_tokens = meter.create_histogram(
    "agent.task.tokens",
    unit="{token}",
    description="Total model tokens used by one agent task",
)

def pseudonymous_actor(actor_id: str) -> str:
    return hashlib.sha256(actor_id.encode("utf-8")).hexdigest()[:16]

def run_observed_task(
    *,
    request_class: str,
    actor_id: str,
    agent_version: str,
    execute: Callable[[], dict[str, Any]],
) -> dict[str, Any]:
    started = time.perf_counter()
    outcome = "internal_error"
    total_tokens = 0

    bounded = {
        "agent.name": "support-triage",
        "agent.version": agent_version,
        "agent.request.class": request_class,
        "deployment.environment": "production",
    }

    with tracer.start_as_current_span("agent.task", attributes=bounded) as span:
        span.set_attribute("enduser.pseudo_id", pseudonymous_actor(actor_id))

        try:
            result = execute()
            outcome = result["outcome"]
            total_tokens = int(result.get("total_tokens", 0))

            span.set_attribute("agent.outcome", outcome)
            span.set_attribute(
                "agent.termination.reason",
                result.get("termination_reason", "completed"),
            )
            span.set_attribute("agent.step.count", int(result.get("steps", 0)))

            if outcome not in {"completed", "escalated", "safely_refused"}:
                span.set_status(Status(StatusCode.ERROR, outcome))

            return result
        except TimeoutError as exc:
            outcome = "deadline_exceeded"
            span.record_exception(exc)
            span.set_attribute("error.type", "deadline_exceeded")
            span.set_status(Status(StatusCode.ERROR, "task deadline exceeded"))
            raise
        except Exception as exc:
            span.record_exception(exc)
            span.set_attribute("error.type", type(exc).__name__)
            span.set_status(Status(StatusCode.ERROR, "unhandled task failure"))
            raise
        finally:
            duration = time.perf_counter() - started
            dimensions = {**bounded, "agent.outcome": outcome}
            task_counter.add(1, dimensions)
            task_latency.record(duration, dimensions)
            task_tokens.record(total_tokens, bounded)

This code intentionally excludes prompts, answers, raw actor IDs, tool arguments, and document text. It emits stable dimensions suitable for aggregation. Whether safely_refused counts as user success depends on the request class: refusing an unauthorized transfer is correct behavior; refusing a permitted password-reset lookup may be a product failure.

Instrument a tool boundary

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("com.codersarts.support-agent.tools")

def get_case_status(case_reference: str) -> dict[str, str]:
    with tracer.start_as_current_span("tool.get_case_status") as span:
        span.set_attribute("tool.name", "get_case_status")
        span.set_attribute("tool.schema.version", "2")

        if not case_reference.startswith("CASE-"):
            span.set_attribute("tool.outcome", "validation_failed")
            span.set_status(Status(StatusCode.ERROR, "invalid case reference"))
            raise ValueError("Invalid case reference")

        # The downstream client enforces tenant authorization independently.
        response = approved_case_client.get_status(case_reference)

        span.set_attribute("tool.outcome", "completed")
        span.set_attribute("http.response.status_code", response.status_code)
        return response.safe_json()

Do not add case_reference as a metric dimension. If it is required for request-level diagnosis, store a protected pseudonymous reference in a span or audit log under a reviewed retention policy.

Measure Latency as a Budget, Not One Number

End-to-end latency is the sum of multiple waits and computations:

Task latency =
  admission + identity + session/state + planning
  + model calls + retrieval + policy + tools
  + retries/backoff + validation + streaming/delivery

AgentCore's Runtime Latency measures time from receiving a request through sending the final response token. Amazon Bedrock's InvocationLatency covers a model invocation through the last token. For streaming Bedrock operations, TimeToFirstToken measures how quickly the first token arrives. These are related but different service boundaries.

Track the latency metrics users actually feel

Metric	Meaning	Operational use
Time to acknowledgement	client receives confirmation that work started	detects admission and cold-path problems
Time to first token	user sees the first streamed content	interactive responsiveness
Time to first useful result	agent presents evidence or a usable action	better than TTFT for verbose “thinking” output
End-to-end task latency	final verified outcome is returned	SLO and capacity planning
Model latency per call	each model dependency duration	model/provider diagnosis
Retrieval latency	search and reranking time	index/filter/reranker diagnosis
Tool latency	downstream system duration	dependency ownership and timeout tuning
Approval wait	human or policy wait time	workflow design, usually separated from compute SLO
Queue time	time before work begins	concurrency and backpressure

Always inspect distributions

Averages hide the experience that creates support tickets. Track p50, p90, p95, and p99 by:

• agent and endpoint version;

  
  

• request class or intent family;

  
  

• model/inference profile;

  
  

• tool and downstream dependency;

  
  

• Region and environment;

  
  

• streaming versus non-streaming; and

  
  

• success, escalation, refusal, and failure outcome.

  
  
  
  

Do not dimension CloudWatch metrics by raw session, trace, user, prompt, case, or document ID. Those belong in trace or log search. High-cardinality metrics increase cost and make dashboards unstable.

Diagnose slow requests with critical-path analysis

When p95 increases:

1. confirm whether Runtime latency and user-observed latency moved together;

   
   

2. compare time to first token with completion latency;

   
   

3. inspect slow traces by request class and version;

   
   

4. identify the longest critical-path span, not the largest total span count;

   
   

5. check whether steps, retries, or tokens per task changed;

   
   

6. compare model latency with output tokens per second;

   
   

7. inspect downstream throttles, connection pools, DNS, VPC/NAT paths, and timeouts; and verify whether observability export is blocking the request path.

   
   

A latency increase caused by longer, higher-quality answers is different from a latency increase caused by a stuck tool retry. The trace must make that difference visible.

Track Tokens Without Confusing Usage, Quota, and Cost

Amazon Bedrock publishes InputTokenCount, OutputTokenCount, cache-read and cache-write token metrics where applicable, EstimatedTPMQuotaUsage, invocation counts, latency, errors, and throttles in AWS/Bedrock. AWS cautions that estimated TPM usage is approximate and should not be the sole capacity-planning signal because throttling can depend on reservation behavior involving input tokens and configured maximum output.

At the individual call level, the Bedrock response can provide usage counts. Aggregate them into the parent task:

response = bedrock.converse(
    modelId=MODEL_ID,
    messages=messages,
    inferenceConfig={"maxTokens": 700, "temperature": 0},
    requestMetadata={
        "application": "support-triage",
        "environment": "production",
        "feature": "case-summary",
    },
)

usage = response.get("usage", {})
input_tokens = int(usage.get("inputTokens", 0))
output_tokens = int(usage.get("outputTokens", 0))
total_tokens = int(usage.get("totalTokens", input_tokens + output_tokens))

current_span = trace.get_current_span()
current_span.set_attribute("gen_ai.usage.input_tokens", input_tokens)
current_span.set_attribute("gen_ai.usage.output_tokens", output_tokens)

Use only approved low-cardinality requestMetadata. Amazon Bedrock model invocation logs capture this optional metadata along with the model/inference profile, request ID, IAM identity, and token counts. Request metadata supports per-request analysis; it is not an AWS resource tag and does not appear as a per-request billing line.

Token metrics that reveal agent regressions

Track:

• input, output, cached-read, cached-write, and reasoning tokens where exposed;

• tokens per model call;

• model calls per task;

• total tokens per task;

• tokens per successful task;

• tokens by request class and agent version;

• p50, p95, and p99 tokens per task;

• maximum-output-token utilization;

• tool-result and retrieved-context size before model invocation; and

• estimated model cost per successful task.

Tokens per successful task is usually more actionable than tokens per call. An update may reduce tokens per call while adding two unnecessary planning calls.

Use invocation logging deliberately

Bedrock model invocation logging is disabled by default. When enabled for supported bedrock-runtime operations, it can deliver invocation records to CloudWatch Logs and/or S3 in the same account and Region. Depending on configuration, those records can contain request/response bodies not only token counts.

Choose a logging mode by risk:

Mode	Captured	Appropriate use
Metrics only	aggregate count, latency, tokens, errors	default for sensitive workloads with limited diagnostic need
Metadata plus usage	model, request ID, pseudonymous dimensions, token counts	routine production attribution
Sampled redacted content	approved prompt/output sample after redaction	quality investigation and evaluator calibration
Full content in isolated destination	complete supported payloads under strict access and retention	rare regulated audit or incident use case after legal/security approval

Enabling full content “for debugging” can copy customer records, secrets, retrieved documents, and model responses into logs and S3. Treat invocation logging as a data-processing system with classification, encryption, IAM, retention, deletion, residency, access audit, and incident-response requirements.

Example token analysis query

For Bedrock model invocation logs:

fields requestMetadata.application as application,
       requestMetadata.feature as feature,
       modelId,
       input.inputTokenCount as inputTokens,
       output.outputTokenCount as outputTokens
| stats sum(inputTokens) as totalInputTokens,
        sum(outputTokens) as totalOutputTokens,
        avg(inputTokens + outputTokens) as avgTokensPerCall,
        count() as modelCalls
  by application, feature, modelId
| sort totalInputTokens desc

Reconcile estimated per-request cost with Cost Explorer or CUR at the available billing grain. Per-request token-derived cost is an estimate and may not reflect negotiated rates, commitments, provisioned throughput, cache pricing, batch pricing, or free-tier effects.

Build a Failure Taxonomy Before Building Alerts

“Agent failed” is too broad for ownership or remediation. Classify failures at the point where they occur.

Failure class	Examples	Primary owner	Key evidence
Admission and identity	invalid JWT, expired token, denied IAM call, quota rejection	platform/security	API status, identity span, CloudTrail
Runtime	AgentCore system error, timeout, memory pressure, unhealthy container	platform/SRE	Runtime metrics, logs, resource telemetry
Model	throttle, provider error, context limit, invalid structured output	AI platform	model span, AWS/Bedrock metrics, usage
Orchestration	loop, max steps, invalid transition, lost state, bad fallback	agent engineering	graph spans, termination reason, checkpoint logs
Retrieval	no evidence, stale index, unauthorized result, reranking failure	data/RAG team	query span, filters, evidence IDs, freshness
Policy and safety	expected deny, unexpected deny, prohibited allow, guardrail error	security/AI governance	policy decision, guardrail metrics, audit event
Tool	schema validation, timeout, 4xx/5xx, duplicate action, partial commit	application owner	tool span, idempotency key, downstream receipt
Output and quality	unsupported answer, wrong action, malformed JSON, poor refusal	product/AI quality	evaluator, validation event, user feedback
Delivery	stream disconnect, client cancellation, response serialization	app/platform	stream metrics, trace status, client telemetry
Telemetry	missing spans, log delivery failure, clock skew, sampling gap	observability platform	heartbeat, delivery metrics, canary

Separate expected denials from defects

An authorization denial can be correct. A guardrail intervention can be correct. A request validation error can be caused by a broken client. Track outcome and error dimensions separately:

transport_status = success
task_outcome = safely_refused
policy_decision = deny
error_type = none

versus:

transport_status = success
task_outcome = failed
policy_decision = deny
error_type = policy_attribute_missing

If both are counted as generic errors, operators page on healthy security controls and miss policy-deployment defects.

Distinguish retriable from terminal failure

Define retryability in code, not through model judgment:

• throttle or transient network interruption: bounded retry with jitter;

  
  

• invalid tool parameter: repair once if safe, then stop;

  
  

• authorization denial: terminal unless verified identity or approval changes;

  
  

• side-effect timeout after request submission: query operation status using the idempotency key before retry;

  
  

• cross-tenant data detection: stop, quarantine evidence, and trigger security response;

  
  

• model output validation failure: constrained repair or safe fallback;

  
  

• maximum-step breach: terminate and record the repeated node/tool sequence.

  
  
  
  

Every retry should be a span event or child span with attempt number and reason. Otherwise, latency and token spikes appear mysterious.

Design SLOs Around User Outcomes

Infrastructure SLOs and agent-quality objectives should coexist.

Recommended production objectives

Objective	Example SLI	Notes
Runtime availability	eligible invocations completed without platform failure / eligible invocations	exclude only explicitly defined invalid traffic
Task success	verified completed tasks / eligible tasks	requires product-specific outcome definition
Interactive latency	tasks with TTFT below threshold / streaming tasks	measure at the client boundary when possible
Completion latency	completed tasks below request-class threshold / completed tasks	segment short chat and long workflows
Tool reliability	successful or safely resolved tool attempts / valid authorized attempts	separate denies and invalid requests
Quality	evaluated sessions meeting rubric / evaluated sessions	report confidence and sampling scope
Safety	prohibited actions executed	usually a hard zero-tolerance count
Efficiency	successful tasks within token/cost budget / successful tasks	prevents silent economic regression
Observability coverage	eligible tasks with complete required spans / eligible tasks	telemetry must be measurable too

CloudWatch Application Signals can define SLOs from latency, availability, or other CloudWatch metrics and create burn-rate alarms. Be careful with its standard availability interpretation: CloudWatch documents that default Application Signals availability treats non-5xx responses as successful, including 4xx. For an agent, build a custom agent.task.success SLI if authorization, validation, or business failures should count differently.

Alert on error-budget burn, not every bad request

Use fast and slow burn windows:

• a fast-burn page for a severe outage consuming the error budget quickly;

  
  

• a slow-burn ticket for sustained degradation;

  
  

• an immediate security page for prohibited action, cross-tenant exposure, or secret leakage;

  
  

• a model/token anomaly alert when per-success usage changes materially by version;

  
  

• a quality alert when online evaluation falls below its approved threshold; and

  
  

• a telemetry-coverage alert when traces or logs disappear while traffic continues.

  
  
  
  

Composite CloudWatch alarms can reduce noise by combining related conditions. For example, page the agent on-call when task failure is high and invocation volume is meaningful, while creating a separate platform incident when Runtime system errors and multiple agent endpoints degrade together.

Build Three Dashboards, Not One Wall of Charts

1. Executive and product scorecard

Show:

• eligible tasks and verified completion rate;

  
  

• escalation, refusal, correction, and abandonment rates;

  
  

• cost and tokens per successful task;

  
  

• top request classes and adoption;

  
  

• quality/evaluation trend with coverage;

  
  

• critical safety or data incidents; and

  
  

• SLO status and remaining error budget

  
  

2. Live operations dashboard

Show:

• traffic, active sessions, Runtime latency, errors, and throttles;

  
  

• p50/p95/p99 task latency and TTFT;

  
  

• model call latency, errors, throttles, and tokens;

  
  

• Gateway, Policy, Memory, retrieval, and tool dependency health;

  
  

• step count, retry count, loop termination, and queue depth;

  
  

• current deployment/model/prompt versions; and

  
  

• telemetry delivery health.

3. Engineering investigation view

Support filtering by trace ID, session ID, agent/version, request class, outcome, error type, model, tool, and time window. Present a span waterfall, structured exception, sanitized tool events, token breakdown, evidence references, policy decision, and evaluator result.

Dashboards should link from aggregate anomalies to filtered traces. If an operator sees a p99 spike but cannot reach the affected executions in one or two actions, the visualization is decorative.

Useful CloudWatch Queries

The exact fields depend on your logging contract. The following examples assume structured application logs rather than unstructured print() statements.

Find failure clusters by version and type

fields @timestamp, trace_id, agent_version, request_class, outcome, error_type
| filter service_name = "support-triage"
| filter outcome not in ["completed", "escalated", "safely_refused"]
| stats count() as failures,
        count_distinct(trace_id) as affectedTraces
  by agent_version, request_class, error_type
| sort failures desc

Compare task latency and steps across releases

fields agent_version, duration_ms, step_count, outcome
| filter event_type = "agent_task_completed"
| stats pct(duration_ms, 50) as p50,
        pct(duration_ms, 95) as p95,
        pct(duration_ms, 99) as p99,
        avg(step_count) as avgSteps,
        count() as tasks
  by agent_version, outcome
| sort agent_version desc

Find token-heavy successful tasks

fields @timestamp, trace_id, request_class,
       input_tokens, output_tokens, total_tokens, outcome
| filter event_type = "agent_task_completed"
| filter outcome = "completed"
| sort total_tokens desc
| limit 50

Detect missing telemetry

fields @timestamp, event_type, trace_id
| filter event_type in ["agent_task_started", "agent_task_completed"]
| stats count() as events,
        count_distinct(trace_id) as distinctTraces
  by bin(5m) as timeBucket, event_type
| sort timeBucket desc

Validate query functions against the current CloudWatch Logs Insights syntax in your Region and adapt fields to the actual schema. Store approved queries with the service runbook rather than relying on individual operators' console history.

Sampling, Retention, and Privacy

Observability has three competing pressures: diagnostic depth, privacy, and cost. Resolve them with data tiers.

Telemetry tier	Coverage	Content	Retention approach
Service metrics	100%	aggregate numerical signals	long enough for capacity and seasonal analysis
Minimal structured task logs	100%	IDs, versions, classifications, outcome; no content	operational and audit requirement
Normal traces	sampled	metadata and bounded span attributes	shorter diagnostic window
Error/security traces	high or complete capture where legally allowed	redacted evidence and errors	incident and investigation policy
Prompt/output samples	very low, consented or approved	redacted content	shortest justified period
Evaluation dataset	curated	reviewed and labeled examples	governed dataset lifecycle

AgentCore/CloudWatch supports configurable trace sampling. Start with enough coverage to discover normal variance, then tune by volume, risk, and cost. Preserve metrics for all traffic. Maintain a controlled path to retain failures and high-risk events even if ordinary success traces are sampled.

Protect telemetry as production data

Apply:

• allowlisted log fields instead of “serialize the request”;

  
  

• client-side redaction before export;

  
  

• CloudWatch Logs data protection policies for audit and masking;

  
  

• separate log groups by environment and classification;

  
  

• KMS encryption and least-privilege access;

  
  

• protected unmask permissions;

  
  

• retention and deletion policies;

  
  

• access logging and periodic access review;

  
  

• pseudonymous actor and tenant references;

  
  

• no secrets in span attributes, baggage, exception text, or URLs; and

  
  

• synthetic data in lower environments

  
  
  
  

CloudWatch data protection can help detect and mask sensitive data, including at the account or log-group level for AgentCore logs. It is defense in depth. It does not justify sending known secrets or unrestricted prompts to telemetry.

Connect Operational Traces to Agent Evaluation

Metrics tell you that behavior changed. Evaluations help determine whether the behavior is acceptable.

AgentCore Evaluations supports:

• online evaluation for sampled or filtered production sessions;

  
  

• on-demand evaluation for selected spans or traces during investigation; and

  
  

• batch evaluation for regression baselines, pre/post comparisons, and periodic audits.

  
  
  
  

For LangGraph, AWS documents supported instrumentation through opentelemetry-instrumentation-langchain or openinference-instrumentation-langchain, with ADOT carrying telemetry.

Correlate each evaluation with:

• trace and session ID;

  
  

• agent, graph, prompt, model, tool, policy, and knowledge versions;

  
  

• request class and risk tier;

  
  

• expected response, assertions, or tool trajectory when available;

  
  

• task outcome and user feedback;

  
  

• latency, tokens, tool calls, and cost estimate; and

  
  

• evaluator name and version.

  
  
  
  

Do not reduce evaluation to one global average. Segment correctness, faithfulness, tool selection, tool parameter accuracy, refusal behavior, and goal success. A 92% score is operationally meaningless if the missing 8% is concentrated in payment cancellations or one regulated tenant.

For a deeper evaluation program, see How to Evaluate RAG Quality with Amazon Bedrock and the Codersarts LLM Evaluation and Benchmark Engineering service.

A Failure Investigation: Latency and Tokens Double Without More Errors

Imagine a customer-support agent whose error rate is flat after release v24, but p95 task latency rises from 5.1 seconds to 11.8 seconds and estimated model cost nearly doubles.

The investigation should proceed as follows:

1. Confirm impact. The task-latency SLI and client telemetry both moved. It is not a dashboard calculation change.

   
   

2. Segment. The regression affects multi-turn “case summary” requests on v24; single-turn lookups and v23 remain stable.

   
   

3. Compare traces. Runtime admission and tool latency are unchanged. The second model span is longer and its input tokens are 2.4 times higher.

   
   

4. Inspect graph behavior. Step count is unchanged, so the problem is not a new loop.

   
   

5. Inspect context construction. A Memory update now appends the entire conversation summary on every turn and duplicates retrieved case notes.

   
   

6. Check quality. Online evaluator scores are flat; the extra context is not improving outcomes.

   
   

7. Contain. Route the production endpoint back to the prior context-builder version or disable the new memory expansion.

   
   

8. Verify. p95 latency, tokens per successful task, and model span duration return to baseline without reducing task success.

   
   

9. Prevent recurrence. Add a context-size budget, duplication test, p95 tokens-per-task release gate, and a regression case based on the confirmed incident.

   
   
   
   

Nothing in that scenario produced a 5xx error. Traditional uptime monitoring would report healthy while users waited twice as long and the organization paid twice as much.

Production Runbook by Symptom

Symptom	First checks	Likely causes	Safe containment
Runtime errors spike across agents	AgentCore system errors, Region status, endpoint version	service issue or shared deployment defect	fail over only to a tested path; pause promotion
Model throttles rise	InvocationThrottles, estimated quota, retry rate, concurrency	quota/capacity or retry storm	apply backpressure; reduce concurrency; use tested alternate capacity
TTFT rises but completion is stable	streaming model spans, network/client metrics	admission, buffering, guardrail mode, connection path	preserve correctness; tune streaming path
Completion latency and tokens rise	steps, model calls, context size, output length	loop, duplicated memory, prompt expansion	cap steps/context; roll back candidate
Tool latency rises	tool span and downstream SLO	dependency degradation	degrade capability; queue or escalate if safe
Deny rate rises	policy mismatch and no-determining-policy metrics, identity claims	policy or identity rollout	fail closed; revert policy after security review
Task success falls with no technical errors	evaluator, feedback, request mix, version	model/prompt/retrieval drift	pin prior version; expand investigation sample
Token count falls and quality falls	prompt/context/version diff	over-aggressive truncation	restore evidence budget; reevaluate
Spans disappear but traffic remains	Transaction Search, exporter, permissions, log delivery	observability pipeline failure	page telemetry owner; preserve service logs
Possible cross-tenant exposure	trace, evidence refs, actor mapping, policy audit	isolation-key or authorization failure	stop affected traffic and initiate security incident response

Automated remediation should be narrow and reversible. Do not let the agent that caused an operational anomaly autonomously change its own model, policy, memory, or tool permissions.

A 30-Day Implementation Plan

Days 1–5: Define the contract

• Inventory agent entrypoints, models, tools, memory, retrieval, and downstream dependencies.

  
  

• Define request classes, business outcomes, failure taxonomy, and risk tiers.

  
  

• Establish trace/span names and allowed attributes.

  
  

• Select retention, redaction, encryption, and access rules.

  
  

• Define initial availability, latency, task-success, safety, and efficiency objectives.

Days 6–12: Establish telemetry

• Enable CloudWatch Transaction Search and required resource policies.

  
  

• Enable AgentCore observability for Runtime, Gateway, Memory, Identity, Policy, and built-in tools in scope.

  
  

• Add ADOT and framework instrumentation.

  
  

• Propagate W3C or X-Ray trace context through application and tool boundaries.

  
  

• Emit custom task outcome, duration, steps, retries, and token metrics.

  
  

• Configure Bedrock metrics and a reviewed invocation-logging mode.

Days 13–18: Build views and alerts

• Create executive, operations, and engineering dashboards.

  
  

• Add Logs Insights queries and trace links to runbooks.

  
  

• Configure fast/slow burn alarms and critical security alerts.

  
  

• Add telemetry heartbeat and synthetic agent canary.

  
  

• Test alert routing, ownership, and after-hours policy.

Days 19–24: Add evaluation and failure drills

• Create a stratified evaluation set with expected failures and denials.

• Configure online sampling and on-demand investigation evaluation.

  
  

• Run throttle, timeout, malformed tool result, retrieval outage, and policy mismatch drills.

  
  

• Verify idempotency, fail-closed behavior, rollback, and trace completeness.

Days 25–30: Tune and govern

• Measure telemetry volume and cost.

  
  

• Reduce cardinality and unnecessary content.

  
  

• Calibrate alert thresholds against actual traffic.

  
  

• Review access to prompts, traces, logs, and unmask permissions.

  
  

• Convert confirmed incidents into regression tests.

  
  

• Document dashboards, queries, runbooks, escalation, and quarterly review ownership.

Common Observability Mistakes

Logging every prompt and response

It improves short-term debugging by creating long-term privacy, security, retention, and cost risk. Begin with metadata and sampled redacted content.

Treating tokens as cost

Token counts are inputs to a cost model. Runtime compute, Gateway, Memory, retrieval, Guardrails, evaluation, logging, networking, and downstream APIs also matter. Billing adjustments can make token-derived estimates differ from invoiced cost.

Paging on every refusal

Expected safety and authorization refusals are healthy behavior. Alert on prohibited allows, unexpected deny changes, policy mismatches, and user-impact trends.

Using trace IDs as metric dimensions

Metrics need stable, bounded dimensions. Put request-specific identifiers in logs and spans.

Monitoring only the final model call

This hides identity, retrieval, policy, memory, retries, tool side effects, and delivery. The parent trace must represent the task.

Sampling before defining critical events

A low random sample can miss rare security or data-isolation failures. Define must-retain event categories and comply with privacy requirements.

Showing quality without evaluation coverage

A dashboard score without sample size, request mix, evaluator version, and confidence can create false assurance.

Depending on one dashboard for every audience

Executives need outcomes and risk. On-call teams need SLO and dependency health. Engineers need traces and structured evidence. Combine the data model, not the screens.

Production Observability Checklist

Trace design
[ ] One trace represents one user-visible or workflow task.
[ ] Model, retrieval, memory, policy, and tool calls are child spans.
[ ] Trace context crosses API, Runtime, Gateway, queue, and downstream boundaries.
[ ] Session, trace, task, and actor identifiers have distinct meanings.
[ ] Span names and attributes follow a versioned convention.

Metrics and SLOs
[ ] Runtime, model, Guardrails, Gateway, Memory, and tool metrics are distinguished.
[ ] Business task success is measured separately from HTTP success.
[ ] Latency includes TTFT and end-to-end percentiles by request class.
[ ] Tokens, calls, steps, retries, and cost are measured per successful task.
[ ] Fast/slow burn and critical safety alarms are tested.
[ ] Telemetry completeness is itself monitored.

Failure operations
[ ] Failures have stable classes and owners.
[ ] Expected denials are separated from defects.
[ ] Retriable and terminal outcomes are deterministic.
[ ] Side effects use idempotency and downstream receipts.
[ ] Runbooks link symptoms to queries, traces, containment, and escalation.
[ ] Confirmed incidents become regression cases.

Privacy and governance
[ ] Prompt/output logging is an explicit risk decision, not a default.
[ ] Secrets and raw personal data are excluded before telemetry export.
[ ] Metric dimensions are low cardinality and non-sensitive.
[ ] Data protection, encryption, IAM, retention, and deletion are configured.
[ ] Access to traces and unmasked logs is reviewed and audited.
[ ] Evaluation samples and human labels follow the same data governance.

Frequently Asked Questions

What is the difference between monitoring and observability for an AI agent?

Monitoring reports known signals such as latency, error rate, and token usage. Observability lets engineers infer why an unfamiliar failure happened by connecting the task, graph path, model calls, retrieval, policies, tools, versions, and outcome in a traceable data model.

Does AgentCore automatically trace a LangGraph agent?

AgentCore provides service metrics and Runtime spans when observability is enabled. Detailed LangGraph, LangChain, model, and tool activity requires supported framework instrumentation such as the documented LangChain OpenTelemetry or OpenInference instrumentors with ADOT. Add custom spans for business outcomes and organization-specific boundaries.

Where are AgentCore traces stored?

AgentCore telemetry is stored in CloudWatch. Runtime spans can appear in the Runtime log group's spans stream or in the shared aws/spans log group, depending on configuration. CloudWatch Transaction Search must be enabled to use the corresponding trace experience.

Which latency should be used for an AI agent SLO?

Use user-observed task latency segmented by request class. For interactive streaming, add time to first token or first useful result. Runtime and model latency are diagnostic component metrics, not substitutes for the client-visible SLI.

How should token cost be attributed to a user or feature?

For per-request analysis on supported Bedrock runtime APIs, use non-sensitive requestMetadata and model invocation logs, or the captured IAM identity where appropriate. Use native billing attribution such as IAM principal attribution or application inference profiles for invoiced aggregates. Per-request token-derived dollars remain estimates.

Should production prompts and responses be logged?

Not by default. Start with aggregate metrics, structured metadata, and redacted sampled content. Enable broader content logging only when the diagnostic or audit need, legal basis, access restrictions, retention, residency, deletion, and incident controls are explicit.

How much tracing should be sampled?

There is no universal percentage. Keep complete aggregate metrics, sample ordinary success traces based on volume and budget, and define stronger retention for errors or high-risk events where permitted. Revisit sampling after traffic, failure rarity, investigation needs, and telemetry cost are known.

Can CloudWatch detect a bad AI answer?

CloudWatch can surface operational telemetry and evaluation results, but a bad answer must be defined by a deterministic validator, business outcome, user feedback, human review, or evaluator. A successful invocation alone cannot establish correctness.

What should page the AI-agent on-call team?

Page on rapid SLO burn, severe availability loss, prohibited actions, cross-tenant exposure, secret leakage, widespread tool failure, or a critical quality threshold breach. Route slower token drift, expected denial trends, and noncritical evaluator changes to tickets or review queues.

Can an existing observability vendor be used instead of CloudWatch?

Yes. AgentCore emits OpenTelemetry-compatible data, and AWS documents a configuration path for using other observability platforms. Decide whether CloudWatch remains the system of record for AWS service metrics and audit integration, and test context propagation, semantic compatibility, privacy, delivery failure, and cost before switching exporters.

What Production Observability Should Change

The purpose of observability is not to accumulate traces. It should change engineering and operating behavior:

• releases are blocked when task success, safety, latency, or token efficiency regresses;

  
  

• incidents begin with a trace and failure class, not speculative prompt changes;

  
  

• tool and policy owners can see the exact boundary they own;

  
  

• cost is connected to successful outcomes rather than raw calls;

  
  

• privacy teams know what telemetry exists and why;

  
  

• confirmed failures enter the evaluation set; and

  
  

• executives see whether the agent creates reliable value, not just traffic.

  
  

If your agent is being deployed on AgentCore, use this guide alongside How to Deploy a LangGraph AI Agent on Amazon Bedrock AgentCore. For the safety layer, read How to Secure Enterprise AI with Amazon Bedrock Guardrails.

Need Production Observability for an AWS AI Agent?

Codersarts AI Agent Development Services can help instrument and operationalize agents built with Amazon Bedrock, AgentCore, LangGraph, LangChain, custom RAG, and enterprise tools.

We can help with:

• observability architecture and telemetry schemas;

  
  

• OpenTelemetry and ADOT instrumentation;

  
  

• AgentCore and CloudWatch configuration;

  
  

• business-outcome metrics and SLOs;

  
  

• model, token, latency, and cost attribution;

  
  

• failure taxonomies, dashboards, alarms, and runbooks;

  
  

• AgentCore Evaluations and regression suites;

  
  

• privacy-aware logging and trace governance; and

  
  

• production-readiness and incident-response reviews.

  
  

Explore our AI Development Services or LLM Evaluation and Benchmark Engineering.

Discuss your AWS AI agent observability requirement

Bring an architecture diagram, several representative traces or failures, the current AWS deployment, and the business outcomes the agent is supposed to complete. We can turn them into an observable production operating model.

Official Technical References

• Amazon Bedrock AgentCore Observability

• Configure AgentCore observability

• AgentCore generated Runtime observability data

• View AgentCore observability data

• Amazon CloudWatch generative AI observability

• CloudWatch Agent view

• Amazon Bedrock Runtime CloudWatch metrics

• Amazon Bedrock model invocation logging

• Track Amazon Bedrock usage and cost

• Amazon Bedrock Guardrails CloudWatch metrics

• AgentCore evaluation types

• CloudWatch service level objectives

• CloudWatch Logs sensitive-data protection

• OpenTelemetry generative AI semantic conventions