Use choice fashions with Amazon Bedrock AgentCore Evaluations

A number of the questions you ask when evaluating an agent are slender and repeatable. Did this response keep on coverage? Is that this device name secure to approve? Does this reply match the retrieved context? Choice fashions reply precisely that type of query. They take a structured state and a set set of allowed solutions and return a calibrated likelihood over them, quite than free-form textual content whose confidence drifts from run to run. That makes them a robust match for the scoring step in an agent analysis, the place you need a label and a quantity you’ll be able to threshold and belief.

Amazon Bedrock AgentCore Evaluations, a functionality of Amazon Bedrock AgentCore, scores an agent by way of its complete lifecycle, on demand throughout improvement, in batch jobs towards saved classes, and on-line towards manufacturing visitors. Alongside 15 built-in evaluators, AgentCore Evaluations helps customized evaluators that you simply outline as both an LLM-as-a-judge or a code-based evaluator. Customized LLM-as-a-judge evaluators apply your directions and ranking scale with a mannequin in Amazon Bedrock. Code-based evaluators run your personal logic in an AWS Lambda perform, and that logic can name your selection of exterior system, mannequin, or API, together with a choice mannequin.

Choice fashions go well with questions with a set set of solutions that an knowledgeable individual might determine at a look, similar to whether or not the agent refused a request or whether or not a response stayed on matter. After you validate a query by yourself visitors, a choice mannequin could be enough for that call or rating, and you may lengthen the identical strategy to particular harmfulness or stereotyping classes in your coverage. As a result of each backends can reply a number of questions on the identical state in a single request, you need to use them as a screening layer quite than a substitute, working a cross or fail sign throughout your metrics to pick out the turns that warrant deeper evaluation by the built-in evaluators, whose explanations enable you examine what went incorrect.

This put up exhibits the way to join two choice mannequin backends to AgentCore Evaluations by way of code-based evaluators. The primary is Jev, a hosted choice mannequin from TypeSafe. The second is Strands Decider 2B, an open-source choice mannequin you run your self. By the tip of this put up, you’ll know the way to convert the AgentCore evaluator enter into a choice mannequin request, and the mannequin’s possibilities again into an AgentCore analysis outcome. First, although, when do you have to attain for a customized LLM-as-a-judge evaluator, and when does a choice mannequin match higher? The subsequent part breaks down what a choice mannequin is and the trade-offs concerned.

Choice fashions

A call mannequin is a small, specialised mannequin educated to pick out from a set set of choices and return calibrated confidence scores quite than producing free-form textual content. A general-purpose giant language mannequin (LLM) evaluator produces a rating with a rationale. A call mannequin as an alternative produces a likelihood distribution over predefined solutions. That provides you a direct confidence measure you’ll be able to threshold, mixture, and act on programmatically. This makes choice fashions well-suited for analysis duties the place the judgment could be acknowledged as a selection amongst a set set of choices: “Is that this response grounded?” or “Did the agent full the consumer’s aim?”

Choice fashions are most helpful if you need a well-calibrated confidence rating alongside the label, not only a binary judgment. They’re additionally applicable if you need to implement a particular rubric persistently with out counting on a general-purpose LLM to interpret it.

Jev, the hosted choice mannequin from TypeSafe, and Strands Decider, an open-source, 2-billion-parameter choice mannequin from the Strands Brokers crew, each assist three forms of questions. A noul query poses a sure/no query and returns a likelihood between 0 (no) and 1 (sure). A selection query asks which of as much as 255 choices applies, returning essentially the most possible choice together with the complete distribution and a confidence. A rating query asks which of two to 10 ordered ranges applies, returning a probability-weighted rating, the distribution over ranges, and a confidence. Many analysis rubrics match one in every of these varieties. “Is each truth on this response supported by the device outcomes?” is a sure/no query, and “How useful was this response, on a four-level rubric?” is a rating.

Choice fashions versus LLM-as-a-judge

The built-in evaluators in AgentCore right now use a general-purpose LLM to evaluate the agent response towards a rubric and clarify its judgment. That is useful when the analysis requires connecting proof, figuring out lacking info, or working by way of a response in depth in duties the place the reason is as helpful because the rating. Choice fashions should not a substitute for LLM evaluators. They’re an addition that the code-based evaluator framework makes potential. Use them for slender, well-defined checks alongside the built-in evaluators’ broader evaluation.

Answer overview

To indicate how these evaluator varieties work collectively, this put up builds an end-to-end instance of an HR Assistant agent evaluated by deterministic code-based checks, built-in LLM-as-a-judge evaluators, and decision-model evaluators backed by Jev or Strands Decider 2B.

The agent is constructed with Strands Brokers and deployed to AgentCore runtime, a functionality of Amazon Bedrock AgentCore. It solutions worker questions utilizing 5 instruments that return fastened mock knowledge similar to paid time without work (PTO) balances, PTO requests, HR insurance policies, advantages, and pay stubs. We use fastened knowledge on this code pattern to make it easy to confirm evaluator conduct from LLM-as-a-judge and choice fashions. While you moreover have floor reality, you’ll be able to verify whether or not every evaluator catches the best failures. This code pattern helps you arrange the framework for testing choice fashions in comparison with LLM-as-a-judge on your personal use instances.

Determine 1: Finish-to-end structure of the HR Assistant evaluated by code-based checks, built-in evaluators, and decision-model evaluators backed by Jev or Strands Decider

Stipulations

To deploy the pattern, you want the next:

• An AWS account in a Area the place AgentCore Evaluations is obtainable. The pattern defaults to US West (Oregon).
• Amazon CloudWatch Transaction Search enabled within the account.
• Entry to a basis mannequin in Amazon Bedrock.
• Python 3.12 or later, the AWS Command Line Interface (AWS CLI), and Docker.
• Jev requires a TypeSafe API key. Go to typesafe.ai for entry particulars.
• Strands Decider requires a bunch working the Strands Decider server. You’ll be able to run it regionally for testing, or on Amazon Elastic Compute Cloud (Amazon EC2) inside your digital personal cloud (VPC) so the Lambda perform can attain it.

Seeing it in motion

Yow will discover the complete code pattern within the AgentCore samples GitHub repository. Two Python scripts deal with all the things:

• ../utils/deploy.py deploys the HR Assistant to AgentCore runtime.
• consider.py deploys evaluators, invokes the agent, and runs the analysis modes.

Working consider.py with out flags executes 5 steps:

1. Creates the Lambda execution function (AgentCoreLambdaEvaluatorRole).
2. Packages and deploys two deterministic Lambda evaluators: hr-response-length (a trace-level evaluator that checks whether or not the agent’s response stays inside size bounds) and hr-fact-checker (a session-level evaluator that regex-validates HR details towards the identified mock knowledge).
3. Registers each as AgentCore code-based evaluators by way of the management airplane.
4. Runs on-demand analysis with EvaluationClient and OnDemandEvaluationDatasetRunner throughout 5 labeled situations, combining the code-based evaluators with Builtin.Correctness, Builtin.Helpfulness, and Builtin.ResponseRelevance.
5. Creates a web based analysis configuration that repeatedly scores reside classes at 100% sampling.

At this level you’ve got a working analysis pipeline with deterministic checks and LLM-as-a-judge scoring. Choice-model evaluators are opt-in. You’ll be able to add them with a flag, and so they run alongside the bottom evaluators with no adjustments to the prevailing movement. Utilizing the flags --with-jev --jev-secret-arn deploys jev-evaluator and registers three evaluators: JevGroundedness, JevHelpfulness, and JevGoalCompletion. Utilizing the flag --with-decider deploys strands-decider-evaluator and registers the identical three evaluators backed by Strands Decider 2B. Add --decider-ec2 to auto-provision the Amazon EC2 occasion, safety teams, IAM function, and Lambda VPC configuration with out handbook console steps.

Determine 2: The analysis pipeline deployed by consider.py, from deterministic checks by way of built-in and decision-model evaluators

Every decision-model backend contributes three evaluators:

Construct the evaluator with choice fashions

Utilizing the flags --with-jev and --with-decider creates a customized code-based evaluator, the place you’ve got full management of the code you need to run as your evaluator. We create AWS Lambda capabilities the place the choice mannequin is invoked inside an AWS Lambda perform and returns a label or a numeric worth. This Lambda perform is registered because the code-based evaluator in AgentCore Evaluations.

The Lambda that’s registered because the evaluator receives OpenTelemetry spans of the agent. The spans.py module normalizes the shapes OpenTelemetry content material can arrive in after which codecs it appropriately to make the decision to the choice mannequin. For trace-level analysis, solely the focused flip and temporary prior historical past are despatched, whereas the complete dialog is included for session-level.

Determine 3: How the code-based evaluator Lambda adapts OpenTelemetry spans for the choice mannequin and maps the outcome again to AgentCore

The adapter converts the mannequin’s likelihood distribution right into a outcome that AgentCore analysis expects. For a noul query, the likelihood of true turns into the worth and essentially the most possible reply turns into the label. For a rating query, the worth is the probability-weighted anticipated degree scaled to [0, 1]. For a selection query, it’s the weighted common of predefined choice values.

Validate a decision-model evaluator by yourself visitors

While you rewrite a rubric as a choice mannequin query, you’ve created a brand new measurement quite than a duplicate of the previous one. A reframed rubric typically lands stricter or looser than the unique meant, and also you gained’t see that drift till you test it towards actual classes. Earlier than you gate a launch on a decision-model evaluator, validate it the identical approach you’d validate any new measurement, towards a labeled pattern of your personal visitors. The aim of this part is to stroll you thru that validation finish to finish, so that you end with an evaluator you’ll be able to threshold with confidence.

The process has 5 steps.

Step 1. Construct a labeled pattern. Pull a set of saved classes out of your agent’s visitors, with sufficient protection of the failures and edge instances you care about. A pattern that covers solely the completely happy path tells you nothing about the place the evaluator breaks.

Step 2. Get human reference labels. Have three unbiased labelers rating every flip for the property your query measures and take the bulk vote because the reference label. Preserve monitor of which turns had been unanimous and which had been cut up. Though unanimous turns are the clear-cut instances, the split-vote turns are the precious ones, as a result of they reveal ambiguity within the rubric itself earlier than the mannequin ever sees it.

Step 3. Run each evaluators on the identical inputs. Run the decision-model evaluator and the matching built-in evaluator over those self same classes in a single batch analysis. Scoring an identical inputs is what makes the comparability apples to apples.

Step 4. Learn the outcomes by way of three lenses. With the majority-vote labels and the simple and arduous cut up in hand, now you can reply the three questions that let you know whether or not the evaluator is prepared.

• Settlement fee: does it match your people? Measure how typically every evaluator agrees with the majority-vote labels, and do it per query, since one query’s accuracy says nothing about one other’s. Then break the numbers out by straightforward (unanimous) and arduous (split-vote) instances. An evaluator that stumbles solely on the arduous instances is behaving lots like your human labelers did, which is strictly what you need.
• Calibration: do the boldness scores imply one thing? Group turns by the evaluator’s reported likelihood and evaluate every group’s settlement with the labels. Greater confidence ought to monitor greater accuracy. Then test whether or not straightforward instances persistently draw greater confidence than arduous ones. In the event that they do, the mannequin’s uncertainty is monitoring real ambiguity, and you need to use these teams to select a choice threshold per evaluator as an alternative of defaulting to 0.5.
• Disagreement evaluation: the place and why do they differ? Discover the turns the place the 2 evaluators disagree with one another, and test whether or not these are the identical turns your people discovered ambiguous. Learn them. Most disagreements hint again to the wording of the query, and a revised query clears them up. When the errors cluster within the arduous instances, the rubric is the issue quite than the mannequin.

Step 5. Re-run the test when something adjustments. Deal with this as a repeatable test quite than a one-time gate. Any time you reword a query or transfer to a brand new mannequin model, run it once more. Every outcome’s rationalization data the mannequin model that produced it, so you recognize which model a given rating got here from.

Selecting the most effective set of evaluators on your agent

The built-in evaluators stay the inspiration of broad analysis technique in AgentCore Evaluations. They assess the response towards a rubric and its supporting context, and clarify the judgment so builders can examine what went incorrect. For the HR Assistant, a faithfulness analysis can determine a declare that contradicts a device outcome, whereas a helpfulness analysis can level out that the agent answered a stability query however left a requested motion incomplete.

Faithfulness, helpfulness, and correctness can require connecting proof, figuring out lacking info, or working by way of an answer. The preprint JEV-as-a-Decide: Settle for When Assured, Escalate When Uncertain describes a associated distinction as “studying versus deriving.” A slender decision-model test can flag a flip for consideration, whereas the built-in evaluator supplies the broader evaluation and reasoning wanted to overview it. Choice fashions shine brightest if you deal with them as a gate quite than a substitute. The cascade sample works like this. First, run a quick, calibrated choice mannequin on each flip of each session to get an inexpensive cross/fail sign. Then invoke a full LLM evaluator solely on classes the place the sign falls under your threshold.

Determine 4: The cascade sample, the place a calibrated choice mannequin screens each flip and solely low-confidence turns escalate to a built-in evaluator

We advocate the next setup:

1. Configure on-line analysis with the built-in evaluators you depend on, as the usual measure of high quality throughout your brokers.
2. Add decision-model evaluators for domain-specific sure/no and selection inquiries to the identical configuration, together with deterministic code-based evaluators for something you’ll be able to compute precisely.
3. Every week, or on no matter schedule fits your launch cycle, question the net outcomes for classes the place a decision-model test failed or scored close to its threshold, and run batch evaluations on these classes with the built-in evaluators to get explanations.
4. Gate releases on the built-in evaluators and on the decision-model evaluators you’ve got validated towards labeled knowledge, and monitor how typically the 2 agree.

Clear up

To keep away from ongoing expenses, run the cleanup.py that deletes the agent hosted on AgentCore runtime, the evaluators and analysis config, the AWS Lambda capabilities which can be registered because the evaluators, the EC2 occasion that hosts the Strands Decider mannequin, AWS Secrets and techniques Supervisor key and the related log group on CloudWatch.

Conclusion

On this put up, you linked two choice fashions, Jev and Strands Decider, to AgentCore Evaluations utilizing code-based evaluators, and the accompanying code pattern offers you a framework for testing these fashions and seeing the place they add worth as evaluators. The built-in evaluators stay your baseline. Choice fashions add slender, validated checks that include a measure of certainty you’ll be able to threshold and act on. The sensible path ahead is so as to add decision-model checks alongside the built-in evaluators you already run, measure how typically they agree along with your reference labels by yourself visitors, and broaden protection solely when you belief them.

To get began, overview the Amazon Bedrock AgentCore Evaluations documentation, combine customized evaluators utilizing code-based evaluators and take a deeper look into Strands Decider mannequin. Get began with code samples by deploying the pattern evaluators from the Amazon Bedrock AgentCore pattern repository.

Concerning the authors