Build agentic retrieval on a Managed Amazon Bedrock Knowledge Base with CDK

This post builds enterprise agentic retrieval on a Managed Amazon Bedrock Knowledge Base with the AWS Cloud Development Kit (AWS CDK). To see why that’s worth building, it helps to start with where ordinary Retrieval Augmented Generation (RAG) breaks down. Most RAG systems start the same way: they embed the question, fetch the nearest chunks from one index, hand them to a model, and return the answer. One retrieval, one generation. For a large class of questions that’s the right design, and there’s no reason to build anything more complicated.

It fails in a recognizable way. Ask a question whose answer spans two corpora, and a single retrieval returns half of what the model needs, which is worse than returning nothing, because the answer comes back fluent and incomplete. Ask a question where the right source depends on the question, and a single index has no mechanism to choose. Ask a question that needs a second look at the corpus after seeing the first result, and there is nowhere for that second look to happen.

Enterprise agentic retrieval answers all three by putting a reasoning agent in the loop. The agent decides whether to retrieve, which knowledge base to consult, and whether to go back for more before it composes an answer. This is the direction Amazon Bedrock Knowledge Bases have evolved. With the launch of the Managed Knowledge Base, retrieval is no longer a single embed-and-fetch call. It has become an agentic capability: multi-turn planning, tool-driven retrieval, and grounded, cited answers. The AgenticRetrieveStream API offers this as a first-class service feature rather than something you assemble yourself. That works, and it also introduces a genuinely harder operational problem. A single-shot pipeline is a function you can reason about by reading it. An agent that reasons and retrieves in a loop takes a different path on every question. By default you can see neither the path it took nor whether the answer it produced was any good. Observability stops being a nice-to-have and becomes the only way to know the system works.

In August 2024, Build an end-to-end RAG solution using Knowledge Bases for Amazon Bedrock and the AWS CDK deployed the single-shot design as a CDK application: an Amazon Simple Storage Service (Amazon S3) corpus, an Amazon OpenSearch Serverless collection and index, a knowledge base, and a data source, provisioned as CDK stacks whose order you sequenced by hand. Three things have changed since it was published.

1. Amazon Bedrock now offers a managed knowledge base (Type: MANAGED), which owns the ingestion, storage, indexing, and retrieval itself. The OpenSearch Serverless collection and index that post provisions are no longer something you need to create, scale, or pay for with OpenSearch Compute Units (OCUs).
2. Amazon Bedrock AgentCore provides the agentic half, a Gateway that exposes each knowledge base as a Model Context Protocol (MCP) tool, a Runtime that hosts the reasoning agent, and AgenticRetrieveStream, which reranks internally and returns a synthesized, cited answer. Agentic retrieval is available only on the Managed Knowledge Base.
3. The observability and evaluation story is now something you can provision, not something you bolt on. This solution ships two Amazon CloudWatch dashboards spanning seven layers of telemetry, plus two forms of evaluation, on-demand and continuous, from the same deployment.

This post builds that system: an agent that routes across two knowledge bases, retrieves through the Gateway, and returns a grounded, cited answer, with every layer of its behavior instrumented, as a single AWS CDK application in Python. One command deploys all of it, and CDK derives the order.

If you read the 2024 post, the path from an Amazon S3 bucket to the knowledge base will look familiar, and everything after it is new: the vector store is gone, the retrieval is agentic, and roughly half of what the app provisions exists to tell you what the agent did. If you didn’t, this post stands on its own.

The services at play and the path one question takes through them: the agent decides whether to retrieve and which knowledge base to consult, the Gateway exposes each knowledge base as a tool it can call, the Managed Knowledge Bases search and cite, and the loop repeats as many times as the question needs. Everything the agent does along the way lands in Amazon CloudWatch.

Background

Before the walkthrough, it helps to establish three things: what makes retrieval agentic, why the Managed Knowledge Base is the right foundation for it, and why observability and evaluation belong in the design rather than bolted on later.

From RAG to enterprise agentic retrieval

Classic Retrieval Augmented Generation (RAG) does one retrieval and one generation. Enterprise agentic retrieval puts a reasoning agent in the loop: it decides whether and what to retrieve, can retrieve several times to refine, chooses which knowledge base is relevant (semantic routing), and only then composes a grounded answer with citations. With the Managed Knowledge Base, this is a first-class service capability, the AgenticRetrieveStream API performs the multi-turn planning, executes the retrieval, and returns a cited answer, rather than an orchestration layer you build and maintain over a raw vector search. That produces better answers on complex questions, but it also produces a more complex system to operate, which is exactly why observability and evaluation are built in from the start.

Managed compared to do-it-yourself Knowledge Bases

Amazon Bedrock now offers a Managed Knowledge Base (Type: MANAGED): Amazon Bedrock manages the ingestion, storage, indexing, and retrieval for you, including embedding and reranking with service-managed models by default, so there is no vector database to provision, scale, or patch. This differs from a Customer-managed (DIY) Knowledge Base, where you choose and operate the vector store (for example, Amazon OpenSearch Serverless) yourself.

Two differences matter most for this solution:

Capability	Amazon Bedrock managed	Customer-managed (DIY)
Agentic retrieval (AgenticRetrieveStream)	Supported	Not supported
AgentCore Gateway integration	Supported	Not supported
Data store	Auto scaling, fully managed by Amazon Bedrock	You provision, scale, and maintain it
Embedding + reranking	Built-in managed models, no extra cost	You configure them
Infrastructure to manage	None	Vector DB + more

Because agentic retrieval and the AgentCore Gateway connector are available only on the managed knowledge base, it is the natural foundation for enterprise agentic retrieval. It also means Amazon Bedrock manages the storage and indexing for you, so there is no OpenSearch Serverless collection or OCU capacity to provision in this design. The customer-managed option remains the right choice when you want to select and operate your own vector store, for example to reuse an existing OpenSearch Serverless collection or to meet specific data-residency, tuning, or compliance requirements. For enterprise agentic retrieval, the managed knowledge base is what makes the capability possible.

Why observability and evaluation

An agentic system that returns an answer isn’t enough for production. You need to see how it behaves (latency, call volume, token usage), how good the retrieval and answers are, and you need those signals continuously. This solution ships two CloudWatch dashboards spanning seven layers of telemetry, plus two forms of evaluation (on-demand and continuous), all provisioned by the same CDK application. The grounding, citations, and evaluation layers also support responsible use. Cited answers trace back to their source passages so responses stay auditable, and the faithfulness and correctness scores let you catch answers drifting from their sources rather than finding out in production.

Why CDK for this particular solution

Any infrastructure-as-code tool can create these resources. CDK earns its place here for a reason specific to this architecture: the four stacks are densely cross-referenced. The Gateway needs both knowledge base ids. The agent needs the URL and ARN for the Gateway. The dashboards need the KB ids, the Gateway ARN, and the agent runtime id, seven values crossing stack boundaries, each one a dimension on a metric or a target on a connector.

Written as raw templates, that is a set of stack exports and cross-stack lookups kept in agreement by hand, plus a script that deploys in the right order and threads outputs forward, and both drift the moment anything is renamed. In CDK you pass the construct, and the export and the dependency edge are derived from that one act of passing it. The 2024 solution had three stacks and a mostly linear dependency chain. At four stacks with seven values crossing between them, the difference stops being stylistic.

Solution overview

Here is the path a single question takes through the system, end to end. It’s worth a slow look, because the rest of the post walks through it left to right.

The following diagram traces the end-to-end agentic retrieval workflow. A user’s question enters the agent on the AgentCore Runtime, which routes it through the AgentCore Gateway to one of two AgenticRetrieveStream tools, one per managed knowledge base, financial and weather, each ingested from Amazon S3. The selected tool returns a grounded, cited answer, and every step streams to Amazon CloudWatch and AWS X-Ray. The numbered badges map to the following walkthrough.

Figure 2: End-to-end agentic retrieval workflow for a single question

The numbered steps in the diagram map to the workflow, which is as follows.

Step 0 — one-time, at deploy. Before any question is asked, each corpus in Amazon S3 is ingested into its own Managed Knowledge Base: Amazon Bedrock parses, chunks, embeds, and indexes it automatically. This happens once when the stacks deploy, it isn’t part of the per-question path, so it is drawn as a dashed arrow and marked step 0. Every question then follows steps 1–6:

1. A user sends a question to the agent hosted on the Amazon Bedrock AgentCore Runtime. The Runtime auto-instruments the whole reason-and-act loop with OpenTelemetry (OTEL) spans, so the system is observable from the very first call.
2. The agent’s reasoning model plans and routes across knowledge bases. Because each knowledge base is exposed as its own retrieval tool, the agent selects the tool whose topic matches the question, financial or weather, rather than searching one blended index.
3. The selected tool call is brokered by the AgentCore Gateway over MCP, authorized by the agent’s IAM task role and signed with SigV4, which invokes that knowledge base’s AgenticRetrieveStream API on the Managed Knowledge Base.
4. AgenticRetrieveStream does the within-knowledge-base work: it decomposes the question into sub-queries, retrieves iteratively from the data store that Amazon Bedrock manages (ingested from the corpora in Amazon S3), and synthesizes a grounded, cited answer that streams back through the Gateway.
5. The agent checks whether the returned context is sufficient. If not, it loops, routing and retrieving again, before composing its final answer. If so, it returns the cited answer to the user.
6. Throughout, the Runtime emits spans, token counts, and metrics to Amazon CloudWatch and AWS X-Ray, populating the seven observability layers and feeding both the on-demand and continuous evaluation scores.

Internally, that workflow is one CDK app, app.py, that declares four stacks. You don’t choose the order, CDK derives it from the construct references you pass:

kb_stack = KnowledgeBasesStack(app, f"{project}-kb", env=env)

gateway_stack = AgenticGatewayStack(
    app, f"{project}-gateway", env=env,
    financial_kb=kb_stack.financial_kb,  # ← the reference creates the dependency
    weather_kb=kb_stack.weather_kb,
)

agent_stack = AgentRuntimeStack(
    app, f"{project}-agent", env=env,
    gateway=gateway_stack.gateway,
)

DashboardsStack(
    app, f"{project}-dashboards", env=env,
    financial_kb=kb_stack.financial_kb,
    weather_kb=kb_stack.weather_kb,
    gateway=gateway_stack.gateway,
    agent_runtime=agent_stack.agent_runtime,
)

That is the whole wiring. Those four constructor calls are what replace the export/import bookkeeping and the deploy script.

The agent runs a reason-and-act loop: it thinks (a large language model (LLM) call), decides which KB tool to call, reads what came back through the Gateway, and often retrieves again before composing its final, cited answer. Every step is auto-instrumented by the Runtime, so the seven observability layers fill from real traffic.

The seven layers. L1 KB-native metrics, L2 ingestion, L3 agentic retrieval quality, L4 Gateway/MCP metrics, L5 OTEL span tree , L6 token usage, and L7 evaluation scores. Layers 1, 4, and 5 are emitted automatically. Layers 3, 6, and 7 are published as custom metrics by the driver notebook.

Everything tunable lives in one file, config.py:

Changing PROJECT_NAME gives you an independently-named copy of the whole solution, so two deployments can coexist in one account. The driver notebook reads the same constant, so it follows the rename automatically, there are no stack names to keep in sync by hand.

Describe the dataset

The solution ships with two small corpora, bundled in the repository under data/:

• Financial — A synthetic Octank Financial 10-K (octank_financial_10K.pdf, ~198 KB).
• Weather — A real, publicly available U.S. Congressional Research Service report on tornadoes (IF12695, tornadoes_report.pdf, ~560 KB).

About the data: The financial corpus is synthetic. Octank is a fictional company and the 10-K is model-generated sample data (CC0-licensed). It isn’t real financial data. The weather corpus is a real, public-domain U.S. Government work and is a Congressional Research Service report, publicly available at sgp.fas.org/crs/misc/IF12695.pdf (works of the U.S. Government aren’t subject to copyright). To use your own data, drop files under data/ and adjust the prefixes in config.py, no other change is required.

The two corpora are intentionally distinct, so the agent must route each question to the right knowledge base, the semantic-routing story. Because these are Managed Knowledge Bases, we don’t configure chunking, embedding, or an index: on ingestion, Amazon Bedrock parses each PDF, chunks it, embeds it with its managed model, and indexes it automatically.

We use two separate knowledge bases, each exposed through the Gateway as its own retrieval tool, rather than one knowledge base holding both corpora and that choice is deliberate. Giving the agent two named tools makes routing an explicit act of reasoning it can be observed making: the span tree shows which tool it chose for a given question, and the retrieval-quality metrics in Layer 3 stay separable per knowledge base, so financial and weather precision never blur into a single number. A single tool spanning both corpora would collapse that distinction, the routing decision would move inside one index’s ranking, out of view, and every quality signal would be an average across two unrelated domains. Keeping the corpora and their tools separate is what makes the agent’s behavior legible, which is the whole point of the exercise.

In CDK, getting the corpora into S3 is a BucketDeployment:

s3deploy.BucketDeployment(
    self, "FinancialCorpus",
    sources=[s3deploy.Source.asset(str(ROOT / "data" / "financial"))],
    destination_bucket=self.source_bucket,
    destination_key_prefix=KbConfig.FINANCIAL_PREFIX,
)

BucketDeployment is worth knowing about precisely because the alternative is so tempting: without it, the custom resource that triggers ingestion tends to grow an upload step, and now one resource owns two unrelated jobs. Here it only has to ingest.

Managed ingestion scans, chunks, embeds, and indexes each document automatically: one document indexed per knowledge base, zero failures.

Figure 3: Managed ingestion results, one document indexed per knowledge base with zero failures

Deploy the solution

This section walks through deploying the whole solution with one command. After the prerequisites are met, cdk deploy --all provisions all four stacks in CDK’s derived dependency order and threads every cross-stack value through automatically.

Prerequisites

• An AWS account with permissions for Amazon Bedrock, AgentCore (Runtime + Gateway), AWS Identity and Access Management (IAM), CloudWatch, AWS X-Ray, Amazon Elastic Container Registry (Amazon ECR), AWS CodeBuild, Amazon S3, AWS Lambda, and AWS CloudFormation.
• The agent’s model available in the account (default us.anthropic.claude-haiku-4-5-20251001-v1:0).
• CloudWatch Transaction Search enabled, so OTEL spans land in aws/spans for Layers 5–7.
• Node.js 18+ and the CDK CLI (npm install -g aws-cdk), plus Python 3.13+.
• The account and region bootstrapped for CDK (cdk bootstrap), once per account/region.
• No local Docker, the agent image is built by CodeBuild (ARM64).

One command

To deploy the solution, clone the repository, create a virtual environment, install the dependencies, and run cdk deploy --all. Run the following commands from your terminal:

git clone https://github.com/aws-samples/amazon-bedrock-samples.git
cd rag/managed-knowledge-bases/07-IaaC/managed-kb-observability-cdk/
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cdk bootstrap  # once per account/region
cdk deploy --all

The first two commands clone the sample and move into the project directory. The next two create and activate a Python virtual environment and install the CDK app’s dependencies. cdk bootstrap prepares the account and region for CDK (you only need it once per account/region), and cdk deploy --all provisions all four stacks.

Account and region come from your AWS profile (CDK_DEFAULT_ACCOUNT / CDK_DEFAULT_REGION), so the app deploys with no edits.

Region availability: The managed knowledge base, agentic retrieval (AgenticRetrieveStream), and Amazon Bedrock AgentCore (Runtime and Gateway) aren’t available in every AWS Region. Before you deploy, confirm that all of these are offered in your target Region, and set ACCOUNT_REGION in config.py accordingly. The default is us-west-2. For current availability, see the Amazon Bedrock and Amazon Bedrock AgentCore documentation and the AWS Regional Services List.

There is no deploy script in this version, and that is the point: cdk deploy --all walks the dependency graph it derived from app.py, deploys the four stacks in order, and passes every cross-stack value automatically. The terminal reports each stack as it completes, bmkb-obs-cdk-kb [1/4] (the default PROJECT_NAME), then gateway [2/4], then agent [3/4], then dashboards [4/4] and that order is derived, not declared: it falls out of the construct references between the stacks, so adding a fifth stack that consumes an existing output reorders the deploy on its own.

Stack -agent builds the agent container with CodeBuild, so allow roughly 8–10 minutes for that stage. The other three are quick. Each stack prints its outputs on completion, which is where you pick up the knowledge base ids, the Gateway URL, the runtime ARN, and both dashboard URLs. When the deploy finishes, all four stacks are complete. The following screenshot shows the AWS CloudFormation console after a successful cdk deploy --all.

Figure 4: AWS CloudFormation console after a successful deploy, with all four stacks CREATE_COMPLETE

On the console, all four stacks, kb, gateway, agent, and dashboards, report CREATE_COMPLETE, each with its [n/4] position in the derived dependency order shown in its description.

A note on long-running resources

Two things in this solution take far longer than the 15-minute ceiling of an AWS Lambda function allows for naive polling: the KB ingestion sync and the CodeBuild image build. The AWS CDK Provider framework handles this with an on_event / is_complete split, where one handler starts the work and a second is polled until it reports done.

cr.Provider(
    self, "IngestSyncProvider",
    on_event_handler=on_event,        # StartIngestionJob, return the job id
    is_complete_handler=is_complete,  # GetIngestionJob, report IsComplete
    query_interval=Duration.seconds(30),
    total_timeout=Duration.minutes(30),
)

Nothing blocks, the response protocol back to CloudFormation is owned by the framework rather than hand-written into each handler, and the 30-minute budget is a property rather than something to engineer around.

Launch and test

Open notebooks/01-drive-and-observe.ipynb and run it top to bottom (pip install -r requirements-notebook.txt first). It reads the stack outputs, then sends topic-specific prompts to the deployed agent, one session per KB, so you can see both the answers and the semantic routing in one place: financial questions are answered from the Octank 10-K and weather questions from the tornadoes report, each routed to the matching AgenticRetrieveStream tool automatically.

The notebook discovers everything from the stack outputs, so there is nothing to paste in by hand:

# Stack names are derived from the same PROJECT_NAME the CDK app uses, so renaming the
# project in config.py cannot leave this notebook pointing at stale stacks.
from config import EnvSettings

PROJECT = EnvSettings.PROJECT_NAME
STACKS = {suffix: f'{PROJECT}-{suffix}'
          for suffix in ('kb', 'gateway', 'agent', 'dashboards')}

def outputs(stack_name):
    s = cfn.describe_stacks(StackName=stack_name)['Stacks'][0]
    return {o['OutputKey']: o['OutputValue'] for o in s.get('Outputs', [])}

The notebook imports the same config.py the app does, so the two can never disagree about what the stacks are called.

The driver sends per-KB prompts through the deployed agent. Financial questions route to the financial KB and weather questions to the weather KB, each with a grounded, cited answer.

Figure 5: Per-KB prompts routed to the matching knowledge base, each returning a grounded, cited answer

The notebook also computes the reference-free Layer 3 retrieval-quality signals from what the agent actually retrieved and cited, no ground truth, no extra judge, and publishes them to CloudWatch.

Figure 6: Reference-free Layer 3 retrieval-quality signals per knowledge base

Observe: The seven-layer dashboards

The two dashboards are where the seven layers become visible. The stack creates both dashboards, and driving traffic populates them. The following screenshot shows the two dashboards as they appear in the CloudWatch console.

Figure 7: The two dashboards in the CloudWatch console, end-to-end agentic observability and per-KB observability

Both dashboards appear in the CloudWatch console: one for end-to-end agentic observability and one for per-KB observability (BMKB, Bedrock Managed Knowledge Base). The rest of this section walks through what each layer shows.

In this version the dashboards are built from typed objects rather than a JSON string. A metric is a Metric, a widget is a GraphWidget, and the layout is a list of Rows:

cw.GraphWidget(
    title="L4 — Gateway metrics (MCP)",
    width=12, height=6, period=PERIOD, statistic="Sum",
    left=[
        self._gateway_metric("Invocations", gateway_arn, "tools/call Invocations"),
        self._gateway_metric("SystemErrors", gateway_arn, "SystemErrors"),
        self._gateway_metric("UserErrors", gateway_arn, "UserErrors"),
        self._gateway_metric("Throttles", gateway_arn, "Throttles"),
    ],
)

The practical benefit isn’t fewer lines, dashboards_stack.py runs to 400 lines, more than the equivalent DashboardBody JSON would. It is that a misspelled property or an invalid widget combination fails at cdk synth, on your laptop, in about a second. The same mistake inside a DashboardBody string deploys successfully and renders as a broken widget on the AWS Management Console, where you find it by looking at it.

Because a Row accepts any widget, the documentation is part of the layout rather than a wiki page that drifts. Each dashboard interleaves TextWidgets with the graphs:

[cw.TextWidget(width=24, height=7, markdown=self._l3_explainer())],
[
    self._quality_widget("L3 · Financial KB — agentic quality (reference-free, 0–1)", fin_id),
    self._quality_widget("L3 · Weather KB — agentic quality (reference-free, 0–1)", wx_id),
],

_l3_explainer() returns a markdown table defining each ratio and how to read it, so the widget that explains grounded_coverage ships in the same commit as the widget that plots it, and a reader who opens the dashboard cold doesn’t need to guess what a reference-free metric is.

Dashboard A — agentic observability (end to end). The board opens with an explainer of how each layer counts a different thing (for N queries you’ll see ~N agent invocations, ~2N retrievals, ~3N LLM calls, and ~5N Gateway MCP operations, the agentic loop made visible), then shows KB metrics (L1), Gateway metrics and latency (L4), token usage (L6), and because the agent uses AgenticRetrieveStream, which returns a synthesized, cited answer rather than per-chunk scores, the reference-free Layer 3 quality signals, defined inline and plotted per KB.

On Dashboard A, the Layer 3 reference-free quality signals are plotted per knowledge base. First, the financial KB. The following chart plots its retrieval_utilization, grounded_coverage, and duplicate_rate.

Figure 8: Financial KB Layer 3 quality, plotting retrieval utilization, grounded coverage, and duplicate rate

The weather KB shows the same three signals, tracked separately so the two domains never blur into one number.

Figure 9: Weather KB Layer 3 quality, tracked separately from the financial KB

The two charts plot retrieval_utilization, grounded_coverage, and duplicate_rate for the financial KB (left) and the weather KB (right), so each knowledge base’s retrieval quality stays separable rather than averaged into one number.

Dashboard B — per-KB (BMKB) observability. The operational signals that also determine spend, per KB: index size (from S3 source bytes), retrieve volume, agentic tool-calls, session token usage, and generation token usage by model, each with an inline explanation.

The following screenshots show Dashboard B’s four metric widgets, populated after a run. The first widget reports index size, derived from S3 source bytes, per knowledge base.

Figure 10: Dashboard B index size per knowledge base, derived from Amazon S3 source bytes

The second widget counts retrieve calls per knowledge base.

Figure 11: Dashboard B retrieve-call count per knowledge base

The third widget counts agentic tool-calls through the Gateway. This one is shared across both KBs rather than split.

Figure 12: Dashboard B agentic tool-call count through the AgentCore Gateway, shared across both knowledge bases

The fourth widget reports session token usage per knowledge base.

Figure 13: Dashboard B session token usage per knowledge base

The four widgets show index size (from S3 source bytes), retrieve-call volume, agentic tool-calls through the Gateway, and session token usage. Each is broken out per knowledge base except the Gateway tool-call count, which is shared across both.

One dimension detail is worth calling out, because the typed API won’t catch it for you: the AWS/Bedrock/KnowledgeBases namespace expects a prefixed id, knowledge-base/<id>, while the solution’s own BMKB/* namespaces use the bare id. Both are only strings as far as CDK is concerned.

# L1 — AWS/Bedrock/KnowledgeBases wants the prefixed form.
dimensions_map={"Operation": "Retrieve", "KnowledgeBaseId": f"knowledge-base/{kb_id}"}

# BMKB/* — our own namespaces use the bare id.
dimensions_map={"KnowledgeBaseId": kb_id}

Layer 5: The span tree

Layer 5 is the raw OTEL span tree in aws/spans, and it is the layer that shows you the agentic loop directly. It is worth reading carefully, because the nesting isn’t obvious: one LLM call produces three spans. The Strands invoke_agent span wraps the turn, an internal chat span wraps the model call, and a chat <model> span of kind CLIENT is the actual call, and all three repeat the same gen_ai.usage.* token attributes.

The following screenshot shows the span tree in aws/spans, with the three-span nesting visible in the top rows.

Figure 14: Layer 5 span tree in aws/spans, showing the three-span nesting of a single LLM call

Reading the top three rows: an invoke_agent Strands Agents aggregate, then an internal chat span, then a CLIENT chat span, all reporting an identical 7,299 input and 268 output tokens. That is the same LLM call appearing at three levels of the tree, which is why aggregation needs care.

That structure matters the moment you aggregate. To count each LLM call once, filter to the CLIENT leaf:

fields attributes.gen_ai.request.model as model,
       attributes.gen_ai.usage.input_tokens as in_tok,
       attributes.gen_ai.usage.output_tokens as out_tok
| filter ispresent(in_tok) and kind = 'CLIENT' and name like /^chat /
| stats sum(in_tok) as input_tokens, sum(out_tok) as output_tokens,
        count(*) as llm_calls by model

Without that filter the same query reports three times the tokens and three times the calls.

Applying that CLIENT-leaf filter, the following screenshot shows generation token usage by model with each LLM call counted once.

Figure 15: Generation token usage by model, with each LLM call counted once

With the filter in place, the run shows 14 LLM calls totaling 60,472 input and 2,709 output tokens. Without it, the same query would report three times the calls and three times the tokens.

Evaluate: On-demand and continuous

Quality is measured two ways, and both are provisioned by the app.

On-demand: The driver notebook calls AgentCore There is no deploy script in this version, and that is the point: (LLM-as-judge) over each session’s spans for built-in evaluators, Correctness, Faithfulness, Tool-Selection Accuracy and publishes the scores to CloudWatch, where they appear as Layer 7 on Dashboard A.

The following screenshot shows Layer 7 on Dashboard A: on-demand evaluation scores per knowledge base, with the Layer 5 OTEL span table on the same dashboard.

Figure 16: Layer 7 on-demand evaluation scores per knowledge base, beside the Layer 5 span table

Layer 7 shows the on-demand evaluation scores per knowledge base, with the Layer 5 OTEL span table alongside, so the scores and the spans that produced them read together on one board.

Continuous (online): The agent stack also provisions an AWS::BedrockAgentCore::OnlineEvaluationConfig that samples live sessions and scores them automatically, no notebook run required. The configuration itself is visible in the Bedrock AgentCore console under Assess → Evaluations → Evaluation configurations. Results land in the log group /aws/bedrock-agentcore/evaluations/results/<config-id>, and every score is also published as a CloudWatch metric in the Bedrock-AgentCore/Evaluations namespace, keyed by evaluator name and configuration id, so you can graph the scores or alarm on them without parsing logs.

The following screenshot shows the online evaluation configuration in the AgentCore console, under Assess, then Evaluations, then Evaluation configurations.

Figure 17: The online evaluation configuration in the AgentCore console, with four built-in evaluators at 100 percent sampling

The configuration lists four built-in evaluators, Tool-Selection Accuracy, Faithfulness, Correctness, and Response Relevance, set to 100 percent sampling and shown as ACTIVE and enabled.

Because the scores are ordinary CloudWatch metrics, they graph like anything else, no log parsing, and the same widget could carry an alarm. The following screenshot shows those four evaluator scores graphed as ordinary CloudWatch metrics.

Figure 18: The four continuous evaluator scores graphed as ordinary CloudWatch metrics

The four evaluator scores appear as metrics in the Bedrock-AgentCore/Evaluations namespace, each keyed to the evaluation configuration id and the agent’s service name, so you can graph them or alarm on them without parsing logs.

Online results only arrive after a session closes. Scoring runs on the completed session, so the results log group stays empty for roughly the session idle timeout (~30 minutes) after you drive traffic. An empty group right after a run is expected, not a misconfiguration. Each result event carries session.id, so it joins to every other layer. The on-demand path in the notebook has no such wait, which is why both are here.

Across the four built-in evaluators on this run, the continuous path scored 1.00 on Correctness, Faithfulness, Response Relevance, and Tool-Selection Accuracy, five scored traces per evaluator, each result carrying a written explanation.

Sampling and cost. This app sets SAMPLING_PERCENTAGE = 100 purely for the blog experiment, so every session is scored and the results are immediately visible. This isn’t a production recommendation. Online evaluation invokes an LLM-as-judge per sampled session, so cost scales with the sampling rate and traffic volume. For a real deployment, choose a sampling percentage that fits your quality-monitoring needs and budget, and align the configuration with your organization’s own policies and cost-governance requirements before enabling it. The rate is a single constant, EvalConfig.SAMPLING_PERCENTAGE, in config.py.

Clean up

When you’re finished, remove the solution so it stops incurring charges, since stored KB data, the agent image in ECR, and the still-active online evaluation all cost money while the stacks sit idle. One command tears everything down:

cdk destroy --all

CDK deletes the stacks in reverse dependency order on its own, dashboards, then agent, then gateway, then kb: the same graph it derived for the deploy, walked backward. There is no teardown script to keep in agreement with the deploy script, which is one of the more quietly valuable properties of modelling the dependencies once.

Two removal behaviors are worth knowing, because both replace hand-written cleanup code:

self.source_bucket = s3.Bucket(
    self, "SourceBucket",
    removal_policy=RemovalPolicy.DESTROY,
    auto_delete_objects=True,  # replaces a list_object_versions delete loop
)

self.repository = ecr.Repository(
    self, "AgentImageRepo",
    removal_policy=RemovalPolicy.DESTROY,
    empty_on_delete=True,  # replaces an image-delete loop
)

Both are set for sample hygiene, for anything you care about, switch the bucket to RemovalPolicy.RETAIN.

The CDK bootstrap stack (CDKToolkit) and its staging bucket aren’t removed. They are shared account-wide infrastructure that other CDK apps may depend on.

What changed since the single-shot solution

Both this post and the 2024 solution are CDK applications in Python, deployed with cdk deploy. The authoring tool is the constant here, what changed is the architecture underneath it, and most of the change is subtraction:

The trade is real in both directions. A single index answering single-shot questions is cheaper per query, simpler to reason about, and entirely sufficient when one corpus holds the answer. Reach for this design when it doesn’t.

The CDK constructs doing the work

Four constructs carry most of the weight in this app, and they are worth naming because each replaces something people otherwise hand-write:

• Provider – For custom resources that outlive the 15-minute ceiling of an AWS Lambda function. The on_event / is_complete split means nothing sleeps in a poll loop, and the response protocol back to CloudFormation is the framework’s job rather than a vendored urllib callback in every handler.
• BucketDeployment – Uploads the corpora, so the ingestion custom resource only ingests.
• auto_delete_objects=True and empty_on_delete=True, replace the hand-written list_object_versions and image-delete loops that a stack otherwise needs to be destroyable at all.
• repository.grant_pull_push(project) – One line for the eight ecr:* actions CodeBuild needs, spelled correctly by construction.

And typed dashboards, from the Observe section: Metric / GraphWidget / Row objects instead of a DashboardBody JSON string, which moves an invalid widget combination from “renders broken on the console” to “fails at cdk synth.”

Two things CDK doesn’t change, and both are worth knowing before you start:

• Free-form JSON stays camelCase. connector_parameters and parameter_values are passed through verbatim, so the AWS CDK snake_case to PascalCase conversion does not apply inside them. Keys like bucketName and inclusionPrefixes must be spelled exactly as the API expects, and nothing will tell you otherwise until the resource fails.
• String-typed values are still strings. The knowledge-base/<id> versus bare <id> dimension difference from the Observe section is invisible to the type system. Typed constructs check structure, not semantics.

Neither is a reason to prefer one tool over the other. They are the boundary of what “typed” buys you, and knowing where that boundary sits is most of what makes CDK pleasant instead of surprising.

Conclusion

We built a complete enterprise agentic retrieval solution on a Managed Amazon Bedrock Knowledge Base and AgentCore: multi-KB semantic routing, a reasoning agent, seven layers of observability, and both on-demand and continuous evaluation, and deployed all of it as a single AWS CDK application.

Two things are worth taking away. The managed knowledge base removed infrastructure rather than adding it: no vector store to provision, no index to create and wait on, no OCU line item accruing while the system sits idle. It is also the knowledge base type that supports agentic retrieval and the AgentCore Gateway connector, as described in the Amazon Bedrock documentation. And once retrieval became agentic, observability stopped being optional. Roughly half of what this app provisions exists to answer what the agent actually did and whether the answer was any good, which isn’t overhead on an agentic system. It’s the part that makes one operable.

The CDK authoring model earned its place on the seams. Seven values cross stack boundaries in this architecture, and every one of them is a construct reference rather than an export/import pair, which is also why there’s no deploy script, no teardown script, and no fixed order to remember.

From here you can point the data sources at your own corpora, place the agent in a virtual private cloud (VPC), tune the online-evaluation sampling rate to your budget and policies, or add more knowledge bases to the router. To learn more, see the Amazon Bedrock detail page and the Amazon Bedrock documentation. The app, the driver notebook, and the self-contained utilities are all in the accompanying GitHub repository, so you can clone it and deploy your own copy with one command.

About the authors