Constructing agentic workflows with SageMaker AI and Bedrock AgentCore

A standard problem in constructing agentic workflows is mixing managed basis fashions (FMs) with your personal cost-optimized or domain-specific fashions, with out rewriting your agent framework to do it. On this publish, we present you the way to mix OpenAI-compatible endpoints on Amazon SageMaker AI with Amazon Bedrock AgentCore runtime, a functionality of Amazon Bedrock AgentCore, and its managed deployment. Specialised brokers can collaborate on advanced duties whereas every makes use of the mannequin finest suited to its job. This mixture provides you price optimization, information residency, and mannequin flexibility in a single production-ready structure.

We stroll by way of deploying Qwen 3.5 9B on Amazon SageMaker AI, integrating it right into a Strands Brokers multi-agent system alongside fashions on Amazon Bedrock, and transport your entire workflow to Amazon Bedrock AgentCore runtime. The main focus is on the combination mechanics together with the way to get token-level observability from SageMaker endpoints, which Strands doesn’t present by default.

Answer overview

The structure connects three model-hosting paths by way of a single Amazon Bedrock AgentCore container:

Orchestrator agent (Claude Haiku 4.5 on Bedrock) – Classifies person intent and routes duties by way of World cross-Area inference.
Finances agent (Claude Sonnet 4.6 on Bedrock) – Handles 50/30/20 funds breakdowns with structured Pydantic output.
Monetary evaluation agent (Qwen 3.5 9B on Amazon SageMaker AI) – Inventory evaluation and portfolio building utilizing tool-calling.

Amazon Bedrock mannequin availability varies by AWS Area. See Supported fashions by AWS Area in Amazon Bedrock.

A person request enters the orchestrator agent working contained in the Amazon Bedrock AgentCore runtime. The orchestrator makes use of the brokers as instruments sample from Strands Brokers to route the request to both the funds agent or the monetary evaluation agent. Each specialised brokers name their respective fashions. The funds agent invokes Claude Sonnet 4.6 by way of Amazon Bedrock, and the monetary evaluation agent invokes Qwen 3.5 9B by way of a SageMaker AI real-time endpoint utilizing the OpenAI-compatible API. Outcomes stream again by way of the orchestrator to the person. For the whole supply code, see the accompanying GitHub repository. The next diagram illustrates this structure.

Determine 1: Structure of the multi-agent workflow throughout Amazon Bedrock and Amazon SageMaker AI

Conditions

You have to have the next conditions to comply with together with this publish.

An AWS account with permissions for Amazon SageMaker AI, Amazon Bedrock, and AgentCore.

pip set up sagemaker-core openai httpx strands-agents[otel] yfinance pydantic bedrock-agentcore.

An AWS Identification and Entry Administration (IAM) function with sagemaker:InvokeEndpoint and sagemaker:CallWithBearerToken.
Bedrock mannequin entry for Claude Haiku 4.5 and Claude Sonnet 4.6.
Python 3.12+.

Step 1: Deploy Qwen 3.5 9B on SageMaker AI

Deploy Qwen 3.5 9B utilizing the vLLM Deep Studying Container (DLC), picture vllm:0.22.1-gpu-py312-cu130, on ml.g6e.2xlarge.

area = "us-west-2"
model_id = "Qwen/Qwen3.5-9B"
instance_type = "ml.g6e.2xlarge"  # 1x L40S (48GB VRAM)
num_gpu = 1

# vLLM 0.22.1, Python 3.12, CUDA 13.0, Ubuntu 22.04
inference_image = f"763104351884.dkr.ecr.{area}.amazonaws.com/vllm:0.22.1-gpu-py312-cu130-ubuntu22.04-sagemaker"

env = {
    "SM_VLLM_MODEL": model_id,
    "SM_VLLM_TENSOR_PARALLEL_SIZE": "1",
    "SM_VLLM_MAX_MODEL_LEN": "32768",
}

# Create Mannequin
sm.create_model(
    ModelName=model_name,
    ExecutionRoleArn=function,
    PrimaryContainer={"Picture": inference_image, "Setting": env},
)

# Create Endpoint Config + Endpoint
sm.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ProductionVariants=[{
        "VariantName": "v1",
        "ModelName": model_name,
        "InstanceType": instance_type,
        "InitialInstanceCount": 1,
        "ContainerStartupHealthCheckTimeoutInSeconds": 1200,
        "InferenceAmiVersion": inference_ami_version,
    }],
)

sm.create_endpoint(EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name)

Step 2: Construct the multi-agent system

The OpenAI-compatible API of SageMaker AI expects a bearer token. Tokens expire, so for any long-running agent session you want a method to refresh them on each request. Arrange auto-refreshing bearer tokens with an httpx.Auth subclass:

import httpx
from openai import AsyncOpenAI
from sagemaker.core.token_generator import generate_token

class SageMakerAuth(httpx.Auth):
    def __init__(self, area): self.area = area
    def auth_flow(self, request):
        request.headers["Authorization"] = f"Bearer {generate_token(area=self.area)}"
        yield request

strands_client = AsyncOpenAI(
    base_url=f"https://runtime.sagemaker.{REGION}.amazonaws.com/endpoints/{ENDPOINT_NAME}/openai/v1",
    api_key="sagemaker",
    http_client=httpx.AsyncClient(auth=SageMakerAuth(area=REGION)),
)

Construct utilizing Strands Brokers’ brokers as instruments sample with contemporary agent cases per invocation.

from strands import Agent, software
from strands.fashions.openai import OpenAIModel

qwen_model = OpenAIModel(
    shopper=strands_client, model_id="",
    params={"temperature": 0.7, "max_tokens": 4096, "stream_options": {"include_usage": True}},
)

@software
def financial_analysis_agent_tool(question: str) -> str:
    contemporary = Agent(mannequin=qwen_model, instruments=[...], callback_handler=None)
    return str(contemporary(question))

orchestrator = Agent(
    mannequin=BedrockModel(model_id="international.anthropic.claude-haiku-4-5-20251001-v1:0"),
    instruments=[budget_agent_tool, financial_analysis_agent_tool],
)

Step 3: Deploy to Amazon Bedrock AgentCore runtime

Deploy utilizing the bedrock-agentcore-starter-toolkit. See deploy_agentcore.ipynb for the total deployment pocket book.

from bedrock_agentcore_starter_toolkit import Runtime

agentcore_runtime = Runtime()
agentcore_runtime.configure(
    entrypoint="most important.py", auto_create_execution_role=True,
    auto_create_ecr=True, requirements_file="necessities.txt",
    area="ap-south-1", agent_name="personal_finance_agent",
)

launch_result = agentcore_runtime.launch(
    env_vars={
        "SAGEMAKER_ENDPOINT_NAME": "qwen35-9b-260612-082732",
        "SAGEMAKER_REGION": "ap-south-1",
        "AGENT_OBSERVABILITY_ENABLED": "true",
    }
)

Configure observability for SageMaker endpoints

Amazon Bedrock AgentCore runtime devices your brokers with OpenTelemetry robotically, however that instrumentation doesn’t lengthen equally to each mannequin supplier. Earlier than you possibly can monitor price and latency for the Qwen mannequin on Amazon SageMaker AI, you need to perceive the place the default instrumentation falls brief and the way to shut that hole.

The problem: Invisible token utilization

Amazon Bedrock AgentCore runtime robotically devices brokers utilizing OpenTelemetry. Nevertheless, there’s a essential hole:

Amazon Bedrock mannequin calls get full generative AI spans with token counts robotically. No further work is required.
Amazon SageMaker OpenAI-compatible endpoints (by way of Strands OpenAIModel) don’t get automated token telemetry. The instrumentation doesn’t acknowledge them as generative AI calls.

This implies tokens consumed by the monetary evaluation agent calling Qwen 3.5 9B on Amazon SageMaker are utterly invisible in traces. You can’t monitor price, detect regressions, or debug latency.

Root trigger: Strands’ OTEL integration emits spans for software calls and agent lifecycle occasions, but it surely doesn’t emit gen_ai.chat spans with token attributes for the OpenAIModel supplier. The auto-instrumentation of AgentCore solely acknowledges Amazon Bedrock mannequin inference calls (made by way of boto3) as generative AI operations.

The answer: Customized OpenTelemetry spans

Manually emit a gen_ai.chat span that wraps the Amazon SageMaker agent invocation and extracts token utilization from Strands’ inner AgentResult.metrics.accumulated_usage:

from opentelemetry import hint

tracer = hint.get_tracer("financial_analysis_agent")

@software
def financial_analysis_agent_tool(question: str) -> str:
    """Route funding queries to Qwen on SageMaker with observability."""
    with tracer.start_as_current_span("gen_ai.chat", attributes={
        "gen_ai.system": "openai",
        "gen_ai.request.mannequin": f"qwen3.5-9b ({SAGEMAKER_ENDPOINT_NAME})",
        "gen_ai.operation.title": "chat",
    }) as span:
        fa_agent = Agent(
            mannequin=OpenAIModel(
                shopper=strands_client, model_id="",
                params={"temperature": 0.7, "max_tokens": 4096,
                        "stream_options": {"include_usage": True}},
            ),
            system_prompt=FINANCIAL_ANALYSIS_PROMPT,
            instruments=[get_stock_analysis, create_diversified_portfolio, compare_stock_performance],
            callback_handler=None,
        )
        consequence = fa_agent(question)
        # Extract token utilization from Strands agent metrics
        utilization = consequence.metrics.accumulated_usage
        span.set_attribute("gen_ai.utilization.input_tokens", utilization.get("inputTokens", 0))
        span.set_attribute("gen_ai.utilization.output_tokens", utilization.get("outputTokens", 0))
        span.set_attribute("gen_ai.utilization.total_tokens", utilization.get("totalTokens", 0))
        return str(consequence)

Key element: Strands tracks token utilization internally with keys inputTokens, outputTokens, and totalTokens. This dict is populated provided that the mannequin supplier returns utilization information.

Why stream_options is necessary for vLLM

By default, vLLM doesn’t embrace a utilization chunk in streaming responses. Strands receives textual content chunks however by no means a last utilization object. Consequently, accumulated_usage stays at zero. Including stream_options: {"include_usage": True} tells vLLM to ship an additional last chunk with token counts:

qwen_model = OpenAIModel(
    shopper=strands_client,
    model_id="",
    params={
        "temperature": 0.7,
        "max_tokens": 4096,
        "stream_options": {"include_usage": True},  # Vital for token monitoring
    },
)

With out this parameter, your gen_ai.chat spans report 0 tokens. This defeats the aim of the customized span.

Step-by-step configuration

Activate Amazon CloudWatch Transaction Search (one-time per account or Area):

aws xray update-trace-segment-destination --region ap-south-1 --destination CloudWatchLogs

aws xray update-indexing-rule --region ap-south-1 --name "Default" 
  --rule '{"Probabilistic": {"DesiredSamplingPercentage": 100}}'

Set up Strands with OTEL extras: strands-agents[otel]>=1.0.0.
Set AGENT_OBSERVABILITY_ENABLED=true in your code or env vars.
Use opentelemetry-instrument because the container CMD.
Add stream_options: {"include_usage": True} to OpenAIModel params.
Create customized gen_ai.chat span wrapping the SageMaker agent name.

Instance hint output

{
    "title": "gen_ai.chat",
    "attributes": {
        "gen_ai.system": "openai",
        "gen_ai.request.mannequin": "qwen3.5-9b (qwen35-9b-260612-082732)",
        "gen_ai.operation.title": "chat",
        "gen_ai.utilization.input_tokens": 1391,
        "gen_ai.utilization.output_tokens": 1432,
        "gen_ai.utilization.total_tokens": 2823
    },
    "durationNano": 37237386894
}

Agent trajectory on Bedrock AgentCore Observability dashboard

This hint view reveals the gen_ai.chat span for the Amazon SageMaker AI hosted Qwen mannequin alongside the robotically instrumented Amazon Bedrock AgentCore spans, with token counts now seen for each. Constructing this end-to-end observability surfaced a number of implementation particulars value calling out.

AgentCore observability dashboard trace showing the gen_ai.chat span with input, output, and total token counts for the SageMaker-hosted Qwen model

Determine 2: AgentCore observability hint with token counts for the SageMaker-hosted mannequin

Key learnings

Amazon Bedrock AgentCore auto-instruments Bedrock calls – No further work for Claude or Amazon Nova.
SageMaker OpenAI endpoints want guide spans – Strands doesn’t emit gen_ai.chat spans for OpenAIModel.
Token utilization requires stream_options – vLLM doesn’t ship utilization in streaming by default.
Use consequence.metrics.accumulated_usage – Keys: inputTokens, outputTokens, totalTokens.
AWS X-Ray sampling price issues – Default 1 p.c drops most traces. Use one hundred pc throughout growth.
Recent agent cases per request – Singletons trigger concurrent invocation errors.

Extending the sample

This structure is composable. A couple of instructions to discover:

Swap in fine-tuned fashions: Level SM_VLLM_MODEL to your fine-tuned checkpoint on Amazon Easy Storage Service (Amazon S3). The auth layer, OTEL spans, and AgentCore deployment keep unchanged.
A/B check with inference elements: Deploy base and fine-tuned variants on the identical Amazon SageMaker endpoint. Add a variant attribute to your OTEL span to check high quality in traces.
Price-aware routing: Test question complexity earlier than dispatch. Route easy lookups to Haiku on Amazon Bedrock. Reserve the Amazon SageMaker GPU endpoint for multi-step reasoning duties.

Cleansing up

To keep away from incurring future costs, delete the assets:

agentcore_control = boto3.shopper("bedrock-agentcore-control", region_name=area)
agentcore_control.delete_agent_runtime(agentRuntimeId=launch_result.agent_id)
sagemaker_client.delete_endpoint(EndpointName=ENDPOINT_NAME)
sagemaker_client.delete_endpoint_config(EndpointConfigName=f"qwen35-9b-epc-{TIMESTAMP}")
sagemaker_client.delete_model(ModelName=f"qwen35-9b-{TIMESTAMP}")

Conclusion

On this publish, we confirmed the way to join a self-hosted mannequin on Amazon SageMaker AI to Amazon Bedrock AgentCore runtime, and critically, the way to get full token-level observability from Amazon SageMaker endpoints that Strands Brokers doesn’t instrument by default.

httpx.Auth + generate_token() + AsyncOpenAI – Manufacturing-ready SageMaker authentication inside AgentCore.
Customized gen_ai.chat OTEL span + stream_options: {"include_usage": True} – Full token visibility for Amazon SageMaker endpoints.
consequence.metrics.accumulated_usage – The Strands API for extracting token counts.

To get began, clone the accompanying repository and see OBSERVABILITY.md for the whole reference.