Coding brokers corresponding to Claude Code and OpenAI Codex CLI work in duties. A developer asks for a change, and the agent reads recordsdata, edits code, and runs exams till the work is finished. One process can take just a few mannequin calls or dozens. What the developer pays for is the duty, not one name.
The worth per token alone doesn’t let you know what an entire process prices. The full relies on what number of turns a mannequin wants, how a lot historical past every flip replays, and the way usually a process needs to be retried or by no means finishes. You pay for each try, together with makes an attempt on duties that by no means end. So whenever you route a coding agent’s requests between fashions, choose the routing by value per solved process: the whole value of your runs divided by the variety of runs whose exams cross.
NVIDIA NeMo Switchyard is an open supply library that helps an AI agent select which mannequin handles every request. You outline the fashions that it could actually use and the technique that it follows. Its LiteLLM integration helps two methods. Stage routing reads the latest device exercise and ends in the request historical past, after which picks considered one of two candidate fashions. Random routing picks from the configured fashions, with non-obligatory weights, for site visitors splits and A/B comparisons. Another choice you may discover is LiteLLM, who additionally just lately shipped a local built-in Switchyard stage router as an auto-router technique.
On this publish, you utilize the AWS Cloud Improvement Package (AWS CDK) to deploy LiteLLM, an open supply gateway for big language fashions (LLMs), with the Switchyard routing plugin in your AWS account. LiteLLM calls the fashions on Amazon Bedrock. You then use the pattern’s benchmark runners to measure value per solved process for a routed mannequin group and for single-model baseline teams, with the intention to consider routing in opposition to your personal workloads.
Notice: The pattern installs
switchyard-server0.3.0 from crates.io and runs the printed LiteLLM v1.102.0 picture. NVIDIA paperworkswitchyard-server0.3.0 as a demo and analysis element, and its configuration can change between releases. This pattern makes use of the experimental Switchyard LiteLLM integration and its configuration and APIs could change between releases. That is pattern code, for non-production utilization. Work together with your safety and authorized groups to fulfill your organizational safety, regulatory, and compliance necessities earlier than deployment. Deploy the pattern in a non-production account and take a look at it earlier than counting on it.
Resolution overview
Determine 1 exhibits the structure of the answer. LiteLLM and the Switchyard plugin run in a single container picture on Amazon Elastic Container Service (Amazon ECS) on AWS Fargate, in non-public subnets.
Determine 1: Resolution structure
A request flows by means of these steps. The numbers match the diagram.
- The coding agent sends an OpenAI-compatible request with the mannequin discipline set to
switchyard, the LiteLLM mannequin group that Switchyard routes, and the shared gateway key within theAuthorizationheader. The endpoint is your hostname over HTTPS, with an AWS Certificates Supervisor (ACM) certificates on the load balancer, and solely the IPv4 CIDR vary that you just permit can attain it. With a hosted zone in Amazon Route 53, Route 53 resolves the hostname to the load balancer. - AWS WAF filters the request with IP fame, recognized unhealthy enter, and per-IP rate-limit guidelines. The Utility Load Balancer forwards solely
/v1/chat/completions,/v1/fashions, and/well being/livelinessto the ECS process in a personal subnet, and solutions each different path with 404. - The duty runs one container: LiteLLM with the embedded Switchyard plugin. LiteLLM validates the gateway key and calls the plugin to choose a mannequin. Stage chooses between a succesful mannequin and a lower-cost environment friendly mannequin. It reads the device exercise and ends in the request historical past, so the agent should ship the total dialog historical past. Random chooses from a weighted pool. Neither technique calls a choose mannequin.
- LiteLLM calls the chosen mannequin by means of the Amazon Bedrock
ConverseorConverseStreamAPI, with the duty’s AWS Id and Entry Administration (IAM) position for credentials. The decision goes by means of an Amazon Bedrock Runtime interface endpoint within the Amazon Digital Non-public Cloud (Amazon VPC), so inference site visitors stays on the AWS community. - The response or token stream returns by means of LiteLLM and the load balancer. The
x-litellm-model-nameresponse header names the mannequin that answered. The response physique’smannequindiscipline staysswitchyard.
LiteLLM listens on port 4000 within the container and is the load balancer’s goal, and the duty’s safety group admits solely the load balancer. The trail allowlist retains the LiteLLM admin UI, API docs, and key administration routes unreachable. Amazon Elastic Container Registry (Amazon ECR) holds the gateway picture. AWS Secrets and techniques Supervisor generates the key behind the gateway key, and the container provides the sk- prefix when it begins. The duty definition passes the profile recordsdata in LITELLM_CONFIG_TEXT and SWITCHYARD_CONFIG_TEXT, and the startup script writes them to /tmp earlier than it begins LiteLLM. Logs go to at least one Amazon CloudWatch log group, /, which retains them for 30 days, and Container Insights collects metrics. The VPC spans two Availability Zones, and one NAT gateway supplies outbound entry for picture pulls and different AWS providers.
The plugin helps Stage routing, which wants precisely two candidates, and Random routing, which wants a minimum of two and splits site visitors by weight for A/B exams. Classifier-backed methods, corresponding to Job and Composite routing, want intermediate mannequin calls that this integration doesn’t help. See the Switchyard routing overview for the total catalog.
Conditions
To observe this walkthrough, you want the next:
- A non-production AWS account with Amazon Bedrock entry to Claude Opus 5.5 and NVIDIA Nemotron 3 Tremendous 120B. The pair has been examined in US East (N. Virginia). For different AWS Areas, examine Regional availability by fashions and tool-calling help.
- The AWS Command Line Interface (AWS CLI) v2 with configured credentials and an express AWS Area, in an account and Area which might be bootstrapped for the AWS CDK.
- Node.js 20 or later, npm, Docker with BuildKit, and Python 3.11 or later.
- A hostname that you just personal, with both a public Route 53 hosted zone for its area or an ACM certificates for it within the deployment Area. A brief plain-HTTP take a look at wants neither.
Walkthrough
You outline the mannequin group, configure Stage, deploy and take a look at the gateway, and measure value. Begin by cloning the pattern repository:
Step 1: Outline the mannequin group in LiteLLM
A LiteLLM mannequin group is one public mannequin title with a number of deployments behind it. The pattern retains its configuration in deployment profiles beneath profiles/. Every profile pairs litellm.yaml, which defines the fashions and gateway settings, with switchyard.toml, which controls how Switchyard picks a mannequin. The next excerpt of profiles/stage/litellm.yaml defines the routed switchyard group, two single-model baseline teams, and the plugin registration:
Each candidates share model_name: switchyard, so your shopper sends "mannequin": "switchyard" and Switchyard picks a candidate. Listing the succesful mannequin first and the environment friendly mannequin second, as a result of Stage reads the roles from that order.
The claude-opus-5-5 and nemotron-super-3-120b teams are single-model baselines. Each all the time solutions with its personal mannequin, so you may evaluate the routed group’s value and cross charge with every mannequin alone in the identical deployment. The Stage plugin alone rejects a bunch with out two candidates, so the picture registers a small wrapper, switchyard_gateway.configured.ROUTING_PLUGIN. It applies Switchyard to teams with two or extra distinctive candidates and passes single-model teams by means of. The configuration registers the wrapper twice: to choose the mannequin and to use the request adjustments that Switchyard data.
Each mannequin makes use of bedrock/converse/, so each candidate goes by means of the Amazon Bedrock Converse API, and validation rejects every other route. The LiteLLM request_timeout of 600 seconds matches the load balancer’s 10-minute idle timeout.
The file holds no credentials. LiteLLM reads the gateway key from the setting, and the duty position provides the AWS credentials. By no means add credentials corresponding to aws_access_key_id, aws_secret_access_key, or api_key beneath litellm_params: the stack copies the entire litellm.yaml into the duty definition and the AWS CloudFormation template in plain textual content.
The stack derives the duty position’s Amazon Bedrock permissions from this file. For a worldwide cross-Area inference profile, the foundation-model useful resource makes use of a * Area and a trailing *, so it additionally matches different mannequin IDs with the identical prefix. Assessment the derived coverage earlier than you deploy.
The world. prefix marks a world cross-Area inference profile, and Amazon Bedrock would possibly course of these requests in supported industrial AWS Areas worldwide. If processing location issues, use a geographic profile or an in-Area mannequin ID as an alternative. For Nemotron 3 Tremendous, the profile makes use of the in-Area mannequin ID nvidia.nemotron-super-3-120b, which you’ll swap for any mannequin for testing.
Step 2: Configure Stage routing
Switchyard reads its algorithm and settings from switchyard.toml. The default profile, profiles/stage/switchyard.toml, configures Stage:
The settings work as follows:
picker = "efficient_first": Stage makes use of the environment friendly mannequin when the alerts within the historical past are inconclusive. A brand new request has no device outcomes but, so it goes to Nemotron 3 Tremendous.confidence_threshold: the boldness cutoff for the execution alerts that Stage scores, from 0 by means of 1.recent_window = 3: Stage examines the three most up-to-date device outcomes.only_on_wrong_signal_escalation = true: the Switchyard default. It controls when Stage provides an non-obligatory escalation notice, and this profile units no notice.
Stage additionally accepts non-obligatory handoff notes and per-tier system prompts (escalation_note, deescalation_note, capable_system_prompt, and efficient_system_prompt). The pattern profile units none of them.
For an A/B take a look at, the Random profile in profiles/random/ makes use of the identical fashions with algorithm = "random" and weights = [0.5, 0.5], one weight per distinctive candidate in declaration order. To deploy it, add -c routingProfile=random to the cdk diff and cdk deploy instructions in Step 3.
The stack validates each recordsdata collectively when it synthesizes, and the preflight script runs the identical checks. To examine a profile change by itself, with out AWS calls, run npm run validate:profile -- stage. Invalid settings cease LiteLLM when it begins, and unsupported message shapes fail earlier than inference.
Non-compulsory: Run the gateway regionally
You’ll be able to attempt the gateway in your machine earlier than you deploy something. Docker Compose builds the identical picture and reads the identical profile recordsdata that the stack deploys to Amazon ECS. Run from the repository root:
The important thing that you just export takes priority over the placeholder in .env, so Compose and the smoke take a look at use the identical one. AWS credentials come solely out of your shell and by no means go in .env or the profile recordsdata. Compose publishes the gateway on 127.0.0.1 solely. There’s no load balancer regionally, so the smoke take a look at skips its closed-path checks. The native gateway nonetheless calls Amazon Bedrock, and people calls are billed. If you’re completed, run docker compose down.
Step 3: Deploy the gateway
The gateway serves HTTPS on a hostname that you just personal. Set GATEWAY_DOMAIN to that hostname and HOSTED_ZONE to an present Route 53 public hosted zone that incorporates it. The AWS CDK then creates a DNS-validated ACM certificates and the alias file. From the repository root, set up the dependencies, run the read-only preflight examine, and deploy:
In case your area’s DNS isn’t in Route 53, use an present ACM certificates as an alternative. Set CERTIFICATE_ARN to an ACM certificates for the hostname within the deployment Area, and change -c hostedZone="$HOSTED_ZONE" with -c certificateArn="$CERTIFICATE_ARN" in each instructions. The AWS CDK then creates no certificates and no DNS file. After the deployment, create a CNAME file (or a Route 53 alias) that factors the hostname on the LoadBalancerDnsName output.
The required allowedCidr worth limits which networks can attain the load balancer, and the command units it to your present public IPv4 tackle. Maintain it to your personal /32: everybody who shares an allowed tackle, corresponding to an workplace NAT or a VPN exit, can attain the endpoint. The stack refuses to synthesize with out allowedCidr, with a world-open vary corresponding to 0.0.0.0/0, or with an IPv6 vary.
Preflight checks the configuration, your instruments, and your AWS entry, together with entry to every mannequin within the profile, with out making inference requests. The AWS CDK builds the gateway picture, publishes it to Amazon ECR, and deploys the stack. Deployment prints these outputs:
| Output | What it’s |
| GatewayUrl | The bottom URL. The OpenAI-compatible API is beneath /v1. |
| MasterKeySecretArn | The Secrets and techniques Supervisor secret behind the gateway key. The complete key’s sk- adopted by the key worth. |
| MasterKeyCommand | A command that prints the gateway key. |
| SmokeTestCommand | The smoke take a look at command, with the stack title, Area, and profile stuffed in. |
| RoutingProfile | The energetic routing profile, stage by default. |
| ClusterName, ServiceName | The ECS cluster and repair. The important thing rotation steps use them. |
| LoadBalancerDnsName | Solely with certificateArn: the title to level your hostname at. |
At all times give shoppers the https:// URL. Port 80 stays open solely to redirect to HTTPS, and a shopper configured with http:// has already despatched the important thing and its first request in clear textual content when the redirect arrives.
The load balancer idle timeout is 10 minutes, for lengthy agent responses. The online ACL makes use of the Amazon IP fame record and the recognized unhealthy inputs rule group. It additionally limits every IP tackle to 2,000 requests per 5 minutes. It leaves out the generic physique guidelines for cross-site scripting, SQL injection, and physique measurement, which the supply code in agent requests triggers. The allowedCidr vary and the gateway key nonetheless apply.
Fast take a look at over plain HTTP. With out HTTPS settings, synth stops with an error. For a brief take a look at with no area, you may decide in to plain HTTP on the load balancer’s DNS title:
With allowPlainHttp=true, the AWS CDK prints a warning each time it synthesizes the stack. The gateway key within the Authorization header, your prompts, the device outcomes and supply code that your agent sends, and the mannequin responses cross the community unencrypted between your shopper and the load balancer. Use the opt-in just for a brief take a look at with non-sensitive code from your personal /32. Then rotate the important thing, and run cdk destroy whenever you end. To maneuver the identical stack to HTTPS later, deploy it once more with the HTTPS settings and with out allowPlainHttp.
Step 4: Take a look at the gateway
In the identical terminal that you just used for the deployment, learn the AWS CloudFormation stack outputs and construct the gateway key:
Ship a request to the switchyard mannequin group:
Search for HTTP 200 and a response. The x-litellm-model-name response header exhibits the chosen mannequin as the total LiteLLM mannequin string. A brand new request has no device outcomes to attain, so Stage picks the environment friendly mannequin, bedrock/converse/nvidia.nemotron-super-3-120b.
To examine well being, authentication, and routing, run the smoke take a look at:
The smoke take a look at checks the well being endpoint, that closed paths corresponding to /ui get the load balancer’s personal 404, {that a} request with no key will get 401, and that /v1/fashions lists each mannequin group. For Stage, it checks that an atypical request selects the environment friendly mannequin and {that a} recorded historical past of repeated failures escalates to the succesful mannequin. Then it sends one request to every baseline group. The request and the smoke take a look at make billable Amazon Bedrock calls.
To attempt your personal agent, use $GATEWAY_URL/v1 as its OpenAI-compatible base URL, the gateway key as its API key, and switchyard because the mannequin. Stage wants the total supported request historical past with each request. The upstream adapter helps textual content messages, perform device calls, and textual content device outcomes, and unsupported message shapes fail earlier than inference. The load balancer forwards the Chat Completions path however not the OpenAI Responses or Anthropic Messages paths. Purchasers that use these APIs get a 404 till you add the trail, take a look at it in opposition to the native container, and redeploy.
Step 5: Measure value per solved process
The pattern’s coding runner, bench/run.py, lets a small coding agent record and skim recordsdata, change supply recordsdata, and run the equipped unit exams by means of the gateway. A process is solved when its exams cross on the finish, inside six mannequin turns by default. Listing the duties and begin with slugify on the routed group:
The terminal exhibits the duty, the chosen fashions, the device actions, the take a look at outcomes, and the ultimate code diff. Every run saves its beginning recordsdata, closing workspace, transcript, patch, and outcome beneath bench/outcomes/.
| Job | What the agent should do |
| slugify | Repair a brief perform that builds URL slugs. |
| roman | Write a perform that converts integers to Roman numerals. |
| calc | Write an expression parser with out eval, with priority, an influence operator, features, and error dealing with. |
| lru_ttl | Write a thread-safe least just lately used (LRU) cache with per-entry expiry and actual counters. |
To present the arithmetic parser process room to run exams and try repairs, increase the flip restrict:
After the runs, the runner prints one row per mannequin group, with RUNS, SOLVED, ERRORS, COST/TASK, and COST/SOLVED columns. Value per solved process is the estimated value of all runs on a bunch divided by the runs whose exams handed. The runner costs every flip from the mannequin in its x-litellm-model-name header and the token utilization within the response, utilizing the PRICES desk in bench/run.py. A flip with cached enter, a mannequin with no PRICES entry, or a response with out the header or token counts stays unpriced as an alternative of counting as zero. Add a PRICES entry whenever you add a mannequin to a profile, and examine Amazon Bedrock pricing for present costs. A number of runs on two duties exhibit the workflow. They aren’t sufficient to attract conclusions about value or high quality.
A bunch exhibits n/a with the rationale when any of its runs is unpriced, and COST/SOLVED exhibits - when no run was solved. To see the place the associated fee went, examine the turns: the terminal prints the mannequin that served every flip, and the transcript beneath bench/outcomes/ data it.
To match complete conversations, bench/multiturn.py runs an agentic loop over every coding process, with the mission recordsdata within the first immediate and the take a look at output fed again after each edit. It additionally runs 4 scripted chats of seven turns that alternate easy turns with design, debugging, and code-writing turns, and it replays the total historical past on each flip. Turns with an inexpensive examine (anticipated textual content, a phrase restrict, bullets, or assertions in opposition to the returned perform) file whether or not the examine handed. The design responses aren’t graded.
The --max-conv-cost possibility (default 1 USD) caps every dialog, and --max-spend stops the runner from beginning new conversations. The abstract exhibits, for every mannequin group, the coding conversations that handed, the chat checks that handed, and the estimated value per coding dialog, per chat, and in whole.
The runners execute model-written Python in your machine. The subprocess that runs the exams has no credentials in its setting, nevertheless it isn’t an working system sandbox: the code can learn any file that your consumer can learn and may attain the community. Use a disposable growth setting with no long-lived AWS credentials on disk, and shield the recordsdata beneath bench/outcomes/, which comprise prompts, mannequin responses, device output, and generated code.
Interpret your outcomes
The pattern’s duties are small and illustrative. To determine whether or not routing suits your workload, change or lengthen them with duties that symbolize your personal code base, and hold the next in thoughts as you learn the runner’s output.
- Begin with SOLVED, then learn COST/SOLVED – A bunch that solves fewer duties can present a decrease COST/TASK and nonetheless value extra per solved process, as a result of failed runs add value with out including solved duties. Evaluate the routed group’s COST/SOLVED with every baseline group’s solely after you’ve in contrast what number of duties every group solved.
- Have a look at the onerous duties individually – Combination numbers can disguise a single process that drives the outcome. Break outcomes down by process. Stage decides from failures within the historical past, so it could actually’t route a tough process to the succesful candidate upfront. An escalated flip additionally replays the entire failed try, which may make a restoration dearer than a run that began on the succesful candidate. If just a few onerous duties dominate your value, think about routing them to a devoted mannequin group, or select an environment friendly candidate that solves extra of them.
- Run sufficient repeats – Mannequin outputs fluctuate from run to run. With solely a handful of runs per group, a distinction of 1 or two solved duties might be noise. Improve
--repeatstill the variations you care about are secure, and apply a statistical take a look at, corresponding to Fisher’s actual take a look at for resolve charges, earlier than you act on a spot. - Account for the flip restrict – A low
--max-turnsworth penalizes a mannequin that explores earlier than it edits, as a result of it leaves fewer turns to restore a failing edit. Set the restrict to match how your brokers run in follow. - Account for output caps – If a bunch’s replies hit
max_tokens, the measured value is a decrease sure, and the shortened replies additionally shorten the historical past that later turns replay. Test the transcripts for truncated replies earlier than you evaluate chat prices. - Deal with prices as estimates – The runners estimate value from token utilization and the PRICES desk, not out of your invoice. Immediate caching, batch inference, and pricing adjustments all have an effect on your precise value. Affirm with Amazon Bedrock pricing and your AWS Value and Utilization Stories.
- Take a look at the settings you propose to run – Outcomes apply solely to the fashions, profile settings, and duties that you just measured. In the event you change a candidate, the Stage settings, or swap to the Random profile, measure once more.
Use these readings to decide on amongst three choices for every workload: the routed group, a single-model group, or a profile with completely different candidates or settings. Rerun the identical measurement whenever you change fashions, prompts, or agent habits.
Clear up
To keep away from ongoing expenses, delete the stack. The AWS CDK wants an allowedCidr worth to synthesize the app, however the worth doesn’t matter right here:
CloudFormation deletes the grasp key secret with no restoration window, so copy the important thing first should you want it. The gateway picture stays within the AWS CDK bootstrap repository in Amazon ECR.
Conclusion
On this publish, we confirmed how you can deploy LiteLLM with the NVIDIA NeMo Switchyard routing plugin on AWS, configure Stage and Random routing throughout fashions on Amazon Bedrock, and measure value per solved process for a routed mannequin group alongside single-model baselines. As a result of the whole value of an agentic process relies on turns, historical past, and retries, worth per token alone isn’t a dependable information. Measuring value per solved process by yourself duties provides you the proof to decide on a configuration.
To get began, deploy the pattern repository in a growth account, and measure Stage routing and single-model baselines on duties from your personal workload. To be taught extra, see the Switchyard LiteLLM plugin information and the Amazon Bedrock Person Information. Share your questions within the feedback.
Concerning the authors
