Consider any agent framework with Amazon Bedrock AgentCore Evaluations

AI groups constructing manufacturing brokers face a irritating asymmetry: the range of agent frameworks retains rising, however analysis tooling has not stored tempo. Most analysis methods assume you constructed your agent in a particular means: a particular SDK, a particular massive language mannequin (LLM) shopper, a particular tracing sample. The second you step outdoors that slender compatibility zone, the analysis pipeline breaks.

Groups construct on LangGraph for its workflow orchestration mannequin, on LlamaIndex for its tight integration with retrieval pipelines, and on the OpenAI Brokers SDK when their group standardizes on GPT fashions. They use Google ADK for multi-agent coordination, or the Claude Agent SDK for native Anthropic functionality. They attain for Strands Brokers as a result of its model-driven loop will get a working agent operating on Amazon Bedrock AgentCore in minutes fairly than days. And more and more, they deploy all of those on Amazon Bedrock AgentCore runtime, a functionality of Amazon Bedrock AgentCore. It handles the internet hosting, scaling, reminiscence, and observability infrastructure they might in any other case rebuild for every undertaking.

Amazon Bedrock AgentCore evaluations solves this fragmentation by decoupling analysis from the framework selection. Each main framework helps OpenTelemetry, both natively or by way of a neighborhood instrumentation library. So long as an agent’s telemetry flows by way of OpenTelemetry, the analysis service can rating it, no matter what SDK sits beneath. This put up explains how that works: what telemetry the service reads, the way it decides how you can learn your spans, which attributes carry the analysis information, and the way protection extends to frameworks past the named checklist.

OpenTelemetry because the widespread language

OpenTelemetry is a vendor-neutral instrumentation framework that standardizes how distributed methods emit traces, metrics, and logs. A hint is a tree of spans, and every span represents a single step inside a request: one unit of labor with its identify, timestamps, a set of typed attributes, and non-compulsory span occasions. Spans are exported over the OpenTelemetry Protocol (OTLP) and picked up by a telemetry backend. On AgentCore runtime, that backend is the AWS Distro for OpenTelemetry (ADOT), which routes spans and occasion data to Amazon CloudWatch.

An agent’s execution produces many sorts of span, as a result of an agent does many sorts of labor. A single consumer flip can generate spans for mannequin calls, software calls, doc retrieval from a vector retailer or database, reranking of retrieved outcomes, embedding technology, guardrail checks, prompt-template rendering, reminiscence reads and writes, and the orchestration steps that tie them collectively. The 2 conventions identify these explicitly: the OpenTelemetry GenAI conventions outline operations resembling chat, embeddings, retrieval, execute_tool, invoke_agent, create_agent, plan, and a household of reminiscence operations. OpenInference defines span sorts resembling LLM, TOOL, RETRIEVER, RERANKER, EMBEDDING, AGENT, CHAIN, GUARDRAIL, EVALUATOR, and PROMPT. A manufacturing hint routinely mixes a number of of those.

Out of that full set, the analysis service wants three span roles to reconstruct what occurred in a session and rating it:

• An invoke agent span represents the top-level request-response cycle, which is one consumer flip in a dialog. It carries the consumer immediate and the ultimate agent response.
• Inference spans signify particular person mannequin calls, every carrying the message historical past handed to the mannequin and the mannequin’s reply.
• Execute software spans signify every software the agent known as, carrying the software identify, enter parameters, and outcome.

These three are what the evaluators function on. The service classifies each span it receives, reads the values it wants from these three roles, and passes over the remaining. Richer traces that additionally carry retrieval, reranking, guardrail, or reminiscence spans are dealt with with none particular configuration. These spans merely add context the evaluators don’t require. As frameworks and the conventions add new span sorts over time, this stays forward-compatible: an unfamiliar span variety is context the service skips, not an error.

Session (one runtimeSessionId)
└── Hint (one consumer flip, one trace_id)
    ├── invoke agent span ← learn: consumer immediate + remaining agent response
    ├── inference span ← learn: messages to mannequin + mannequin reply
    ├── execute software span ← learn: software identify + parameters + outcome
    ├── retriever span (context; not required by evaluators)
    ├── inference span ← learn: subsequent mannequin name with software lead to historical past
    └── ... (guardrail, reminiscence, reranker, orchestration spans, and extra)

The service reads the three marked span roles and treats the others as further context.

Totally different frameworks and instrumentation libraries file these three roles utilizing totally different attribute names, nesting constructions, and span-naming conventions. The OpenTelemetry GenAI semantic conventions and the OpenInference specification each outline schemas for recording these three span roles, however they use totally different attribute keys and totally different span-kind vocabularies. AgentCore Evaluations bridges each schemas to the identical outcome.

How AgentCore Evaluations reads your traces

While you run an analysis, on demand or by way of an internet analysis config, the service fetches your agent’s spans and occasion data from CloudWatch and reconstructs the session. A session is grouped by session.id. Inside it, every hint (one trace_id) is one consumer flip. Each flip is made up of the three span varieties described earlier.

The service classifies every span, extracts the values it wants, and palms the reconstructed session to the evaluators. From that time on, analysis is fully framework-agnostic: the identical evaluators, GoalSuccessRate, Correctness, Helpfulness, and customized LLM-as-a-judge, rating each framework identically. That is proven visually within the following determine.

Determine 1: Knowledge move from AgentCore Runtime by way of Amazon CloudWatch to AgentCore Evaluations

You don’t configure any of this. Each OpenTelemetry instrumentation library stamps a scope.identify on the spans and occasion data it produces, and the analysis service makes use of that worth to resolve how you can learn them. The proper dealing with prompts routinely from the instrumentation bundle you put in, with no modifications to your agent code. Every supported framework and its scope identify is listed within the Supported agent frameworks documentation, which at the moment covers Strands Brokers, LangGraph, the OpenAI Brokers SDK, LlamaIndex, Google ADK, and the Claude Agent SDK, most with each OpenTelemetry and OpenInference instrumentation.

Protection reaches past the named frameworks. Any library whose scope identify falls beneath opentelemetry.instrumentation.* (following the OpenTelemetry GenAI semantic conventions) or openinference.instrumentation.* (following the OpenInference specification) is learn by way of a generic path. In apply, supporting a brand new framework is normally solely a matter of putting in a compliant instrumentation bundle: the scope-name prefix is how a library opts in. A scope named mycompany.agent.tracing is not going to be picked up, even when its spans observe the conventions precisely. The prefix is the sign that the instrumentation writer intentionally conformed to a documented schema.

Getting a session evaluated finish to finish comes down to 2 necessities. The primary is grouping: your agent’s spans should carry a session.id attribute that matches the runtimeSessionId you invoked the agent with. That attribute is what lets the service assemble spans into traces and traces right into a session. On AgentCore runtime, ADOT injects this attribute routinely, so no agent code modifications are wanted.

The second requirement is that your information supply consists of the message content material, not solely the spans. For brokers with unified observability (the default for newly created brokers), that is computerized: message content material lives in the identical per-agent log group because the spans, so a single log group is ample. For present brokers nonetheless operating the pre-unified configuration, nonetheless, spans land within the shared aws/spans log group. Message content material is saved individually as correlated occasion data within the agent’s log group. In that configuration, in case your information supply covers solely aws/spans, span classification nonetheless succeeds, however message content material comes again empty, and any evaluator that scores response high quality returns an error.

OpenTelemetry GenAI semantic conventions: Attributes that drive analysis

The OpenTelemetry GenAI semantic conventions outline a schema for spans produced by LLM frameworks. AgentCore Evaluations reads a particular subset of those attributes to categorise spans and extract analysis information. Figuring out which attributes carry that information is beneficial if you end up debugging an analysis or writing customized instrumentation.

Span classification begins with gen_ai.operation.identify:

• invoke_agent marks the invoke agent span, the top-level span per flip, carrying the consumer immediate and remaining response.
• chat marks an inference span. The service reads the message historical past and the mannequin’s reply.
• execute_tool marks an execute software span. The service reads the software identify, enter parameters, and outcome.

When gen_ai.operation.identify is absent, widespread in some LlamaIndex traces, the service falls again to traceloop.span.variety, the place workflow maps to the invoke agent span, software to an execute software span, and llm to an inference span.

Software identification comes from gen_ai.software.identify, and gen_ai.software.name.id is the correlation ID that hyperlinks a software name requested on an inference span to its outcome on the corresponding execute software span. The gen_ai.software.definitions attribute on inference spans carries a JSON-encoded array of software schemas, which the GoalSuccessRate evaluator makes use of to confirm that the instruments the agent known as have been amongst these it declared out there.

Message content material lives in certainly one of two locations, relying on how telemetry was collected. When telemetry is cut up, the content material is within the correlated occasion file physique (physique.enter.messages and physique.output.messages). When it isn’t cut up, the content material stays on the span, as gen_ai.enter.messages or gen_ai.output.messages attributes or as inline span occasions. The service reads from whichever location holds the content material, so that you don’t have to know which path a given deployment took. For the total image of the place content material lives, see Spans, occasion data, and telemetry indicators.

OpenInference conventions: An alternate semantic layer

OpenInference is an open specification maintained by Arize AI and extensively adopted throughout the LlamaIndex, Haystack, and Phoenix ecosystems. It represents LLM operations with totally different attribute names from the OpenTelemetry GenAI conventions. A number of frameworks, together with the OpenAI Brokers SDK with openinference-instrumentation-openai-agents, Google ADK, and the Claude Agent SDK, produce OpenInference spans.

Span classification in OpenInference makes use of openinference.span.variety. The values LLM, TOOL, AGENT, and CHAIN correspond to inference, execute software, invoke agent, and structural container spans respectively.

Inference-span message content material makes use of a flat, listed attribute conference. Enter messages observe the sample llm.input_messages.{i}.message.position and llm.input_messages.{i}.message.content material, the place i begins at zero and increments for every message within the historical past. Software outcome messages carry an extra llm.input_messages.{i}.message.tool_call_id to hyperlink them to the corresponding software name. Output messages observe the identical listed sample, with software calls nested additional:

llm.output_messages.0.message.position = "assistant"
llm.output_messages.0.message.tool_calls.0.tool_call.perform.identify = "get_pto_balance"
llm.output_messages.0.message.tool_calls.0.tool_call.perform.arguments = "{"employee_id":"EMP-001"}"
llm.output_messages.0.message.tool_calls.0.tool_call.id = "call_abc123"

Software schemas on inference spans use llm.instruments.{i}.software.json_schema, the place every worth is a JSON string encoding both a plain perform schema or an OpenAI-style {"kind": "perform", "perform": {...}} wrapper. The service handles each and reads the identify, description, and parameters fields.

Execute software spans use three attributes: software.identify for the ID, enter.worth for the parameters (a JSON-encoded object for multi-argument instruments, or a plain string for single-argument instruments), and output.worth for the outcome. The output.worth format has developed throughout variations of the OpenInference instrumentation library, and the service reads each the older and newer shapes, so you aren’t tied to a particular model.

Evaluating any compliant framework

The named frameworks are lined immediately, however the design is intentionally open-ended. For any framework outdoors that checklist, the 2 generic paths launched earlier present broad protection at any time when the instrumentation follows a documented conference:

• The OpenInference path handles any scope beneath openinference.instrumentation.*. It classifies spans with openinference.span.variety, reads inference content material from the listed llm.input_messages and llm.output_messages attributes, and reads software.identify, enter.worth, and output.worth for software spans. This covers most frameworks that undertake the OpenInference specification, together with ones launched after a given framework was named explicitly.
• The OpenTelemetry path handles any scope beneath opentelemetry.instrumentation.*. It classifies spans with gen_ai.operation.identify, reads message content material from the occasion file physique or span, and makes use of gen_ai.software.identify for software identification.

The scope-name prefix is what routes a library to certainly one of these paths. For groups writing customized instrumentation that needs to be evaluated generically, naming your scope beneath opentelemetry.instrumentation.* or openinference.instrumentation.* is the way in which to decide in.

From agent code to analysis rating: A walkthrough

To make this concrete, the AgentCore samples repository supplies full working examples, as an illustration an HR assistant carried out with the OpenAI Brokers SDK, Google ADK, LlamaIndex, and the Claude Agent SDK. Every deploys to AgentCore runtime and is evaluated with the identical built-in and customized evaluators.

Instrumentation setup requires no specific code in both agent. On AgentCore runtime, AWS Distro for OpenTelemetry (ADOT) discovers the put in instrumentation bundle at startup and prompts it routinely. For the OpenAI Brokers SDK pattern, including opentelemetry-instrumentation-openai-agents to necessities.txt is ample. For the LlamaIndex pattern, opentelemetry-instrumentation-llamaindex does the identical job. The one factor the OpenAI Brokers SDK agent should keep away from is looking set_tracing_disabled(True): the instrumentation hooks into the SDK’s personal tracing pipeline, so disabling SDK tracing silences the analysis spans.

For LlamaIndex particularly, agent construction issues. The agent have to be constructed as a FunctionAgent or ReActAgent (workflow brokers), not as a plain AgentExecutor. Workflow brokers emit a top-level invoke agent span, which anchors the hint. With out it, there is no such thing as a top-level span to reconstruct the flip from, and the session can’t be assembled from inference spans alone.

To reliably ship telemetry to CloudWatch, flush on the finish of the invocation handler earlier than returning the response. AgentCore runtime suspends the execution atmosphere after the handler returns, whereas the OTel SDK buffers telemetry in client-side batch processors that export on a periodic timer. In consequence, unexported information should be sitting within the buffers at return time, and there’s no assure that an export completes earlier than suspension. In apply, lacking telemetry from an omitted flush is the most typical supply of analysis failures. The OpenAI Brokers SDK and LlamaIndex samples implement this with the next sample:

def _flush_telemetry():
    from opentelemetry import hint as _trace
    from opentelemetry._logs import get_logger_provider as _get_lp
    for supplier in (_trace.get_tracer_provider(), _get_lp()):
        flush = getattr(supplier, "force_flush", None)
        if flush:
            flush()

@app.entrypoint
async def invoke(payload, context):
    immediate = payload.get("immediate", "")
    strive:
        outcome = await run_agent(immediate)
    lastly:
        _flush_telemetry()
    return str(outcome)

The flush should cowl each the tracer supplier (spans) and the logger supplier (occasion data): spans and occasion data undergo separate client-side batch processors, so flushing solely certainly one of them leaves the opposite’s buffer untouched.

AgentCore Evaluations makes use of EvaluationClient identically for each frameworks, after a CloudWatch ingestion wait of 90–150 seconds:

from bedrock_agentcore.analysis import EvaluationClient
from bedrock_agentcore.analysis.shopper import ReferenceInputs
from datetime import timedelta

ec = EvaluationClient(region_name=REGION)
# The shopper resolves every evaluator's stage (SESSION, TRACE, or TOOL_CALL)
# routinely, so no stage configuration is required right here.
outcomes = ec.run(
    evaluator_ids=[
        "Builtin.GoalSuccessRate",
        "Builtin.Correctness",
        "Builtin.Helpfulness",
        CUSTOM_RESPONSE_QUALITY_ID,
        CUSTOM_SESSION_COMPLETENESS_ID,
    ],
    agent_id=AGENT_ID,
    session_id=SESSION_ID,
    look_back_time=timedelta(hours=1),
    reference_inputs=ReferenceInputs(
        assertions=ASSERTIONS,
        expected_trajectory=EXPECTED_TRAJECTORY,
        expected_response=EXPECTED_RESPONSES[-1],
    ),
)

The GoalSuccessRate evaluator operates on the session stage and checks whether or not the agent’s tool-call historical past matched the anticipated trajectory throughout all turns. Correctness and Helpfulness are trace-level evaluators that rating every particular person flip. Constructed-in evaluators are configured for the session, hint, software ranges and out there to make use of as templates. For customized evaluators that you simply configure, you outline if it really works on the session, hint or software stage. It’s potential to mix evaluators at numerous ranges right into a single analysis name. The analysis code for the totally different framework samples is structurally an identical. The one variations are the agent names and useful resource IDs.

On-demand and on-line analysis modes

The pattern repository demonstrates each analysis modes, which serve distinct functions in a manufacturing agent workflow.

On-demand analysis is the pure match for steady integration and steady supply (CI/CD) pipelines and regression testing. You invoke the agent to generate a session, then name EvaluationClient.run() in the identical script to attain it. Since you management the invocation, you may provide floor fact by way of ReferenceInputs: anticipated responses, anticipated software trajectories, and behavioral assertions. A pipeline that runs on each pull request can invoke a set set of check prompts, rating them in opposition to floor fact, and fail the construct if any evaluator drops under threshold.

On-line analysis screens stay agent site visitors constantly. You create an OnlineEvaluationConfig as soon as, pointing it on the agent’s CloudWatch log group and specifying a sampling charge. The service picks up new classes routinely as they arrive:

_cp.create_online_evaluation_config(
    onlineEvaluationConfigName="hr_agent_online_eval",
    rule={"samplingConfig": {"samplingPercentage": 25.0}},
    dataSourceConfig={
        "cloudWatchLogs": {
            "logGroupNames": [CW_LOG_GROUP],
            "serviceNames": [OTEL_SERVICE_NAME],
        }
    },
    evaluators=[
        {"evaluatorId": "Builtin.GoalSuccessRate"},
        {"evaluatorId": "Builtin.Correctness"},
        {"evaluatorId": "Builtin.Helpfulness"},
    ],
    evaluationExecutionRoleArn=ONLINE_EVAL_ROLE_ARN,
    enableOnCreate=True,
)

On-line analysis can solely use evaluators that don’t require floor fact, since stay site visitors has no related anticipated responses. The built-in evaluators work in each modes. Customized LLM-as-a-judge evaluators that reference {expected_response} or {assertions} placeholders of their directions are on-demand solely.

On-line analysis outcomes land in a CloudWatch log group at /aws/bedrock-agentcore/evaluations/outcomes/{config_id}. Constructing a CloudWatch alarm or dashboard from that information offers you steady visibility into agent high quality in manufacturing, throughout all frameworks, by way of a single constant interface. As a result of the identical analysis pipeline processes on-demand and on-line site visitors identically, scores from a CI check run and scores from manufacturing site visitors are immediately comparable when the agent’s conduct is steady.

Conclusion

AgentCore Evaluations is framework-agnostic. Whether or not you construct with LangGraph, the OpenAI Brokers SDK, LlamaIndex, Google ADK, the Claude Agent SDK, or Strands Brokers, the identical evaluators apply uniformly in on-demand and on-line modes, as a result of the combination contract is constructed on OpenTelemetry requirements. In case your agent emits spans beneath a acknowledged scope identify, consists of software schemas on inference spans, and surfaces message content material by way of span attributes or occasion data, the analysis service produces scores with none framework-specific code in your facet. For frameworks past the named checklist, following the OpenTelemetry GenAI conventions or the OpenInference specification with the best scope prefix is all it takes. With one analysis suite, throughout each agent that emits telemetry beneath a acknowledged scope, now you can assess high quality constantly.

Able to strive it your self? Get began with the AgentCore Evaluations characteristic samples, and confer with the Supported agent frameworks documentation for full particulars on connecting your framework of selection.

Acknowledgements

Thanks to the AgentCore evaluations engineering workforce led by Shoaib Javed and Qiaoxuan Xue, and the engineers who contributed to this work: Swarnim Singhal, Lefan Zhang, Athul Iddya, Irene Wang, Ritvika Pillai, Aditya Arepalli.

In regards to the authors