As soon as a mannequin is in manufacturing, the questions change. Groups need to know which configuration the mannequin ought to run on now that visitors is actual, learn how to match extra fashions onto GPUs they already pay for, what to do when the following mannequin is simply too giant for one machine, why requests wait in a single Area whereas GPUs sit idle in one other, and why latency rose when no one modified something. This submit takes these questions within the order groups normally meet them.
For HyperPod Inference, it listed extra routing methods, broader help for mannequin serving frameworks, and expanded multi-Area networking. The Inference Gateway delivers the routing, and managed NVIDIA Dynamo is a brand new serving framework. For managed endpoints, the roadmap pointed to deeper integration with the remainder of the AWS AI panorama. Inference suggestions now run from Amazon SageMaker AI Studio, the SageMaker Python SDK, and the coding brokers engineers already use, and endpoints can now route requests throughout Areas and accounts.
To maintain every part concrete, we comply with one hypothetical crew by means of the weblog submit. Within the spring, they launched a buyer help assistant on a 70B-parameter mannequin behind a single SageMaker AI endpoint, utilizing the launches from half 1. By the autumn, they should help extra product strains, undertake a bigger reasoning mannequin, serve clients in a second Area, and hold their GPU invoice flat.
The launches at a look
Select the suitable configuration from wherever you’re employed
The crew picked its first configuration from a benchmark run in April. Since then, they’ve added 4 LoRA (Low-Rank Adaptation) adapters, one for every product line, and the product crew has set a goal of 500 milliseconds for time to first token (TTFT) at peak load. The April configuration was by no means examined in opposition to both change. The engineers who will re-run the evaluation work in notebooks and coding brokers, and few of them use the console.
Inference suggestions launched in April as an API. 4 updates since then make it out there in additional of the locations groups work, and let it reply two more durable questions: what number of adapters an endpoint can carry, and the way a lot concurrency it could take earlier than it misses a latency goal. A brand new agent ability brings the identical experience into coding brokers.
Inference suggestions in SageMaker AI Studio (July 2026)
Weblog: Launching UI for generative AI inference suggestions in Amazon SageMaker AI
The April launch automated occasion, container, and optimization choice by means of APIs. Inference suggestions are actually additionally out there in SageMaker AI Studio as a guided, low-code workflow. In Studio, select Jobs, then Inference optimization. Select a use-case profile (Work together, Generate, Summarize, or Customized), an optimization objective (reduce latency, maximize throughput, or reduce price), and a mannequin from SageMaker JumpStart, Amazon Easy Storage Service (Amazon S3), the Mannequin Registry, or an present SageMaker mannequin.
SageMaker AI benchmarks candidate configurations on actual GPU infrastructure utilizing NVIDIA AIPerf and applies strategies that match the objective, equivalent to speculative decoding for throughput or kernel tuning for latency. You examine the ranked outcomes by TTFT, inter-token latency, throughput, and value, after which deploy the chosen configuration to a real-time endpoint from Studio.
Inference suggestions within the SageMaker Python SDK v3 (August 2026)
Weblog: LLM optimization integration for Amazon SageMaker Python SDK
Benchmarking and proposals beforehand required SageMaker AI Studio or direct Boto3 calls. They’re now a part of the SageMaker Python SDK v3 (model 3.17.0 and later), the identical interface groups use for coaching and deployment, so the workflow suits into notebooks, CI pipelines, and MLOps workflows. You may benchmark a dwell endpoint in opposition to your workload visitors, generate ranked suggestions for a latency, throughput, or price objective, and deploy the top-ranked configuration to a real-time endpoint in a single name.
The SDK additionally helps longer workflows. You may run advice jobs in parallel to match serving frameworks equivalent to LMI and vLLM, and you may resume from a accomplished job in order that deployment occurs in a special pocket book or pipeline stage with out re-running the benchmarks.
LoRA adapter help in inference suggestions (September 2026)
Developer information: Get suggestions for fashions with LoRA adapters
Many groups use LoRA adapters to specialize one basis mannequin for a number of use circumstances. Every adapter saved in GPU reminiscence reduces the variety of concurrent requests the endpoint can serve, and the suitable stability is determined by the occasion sort, the variety of adapters, and the visitors profile.
Inference suggestions now settle for adapters as Mannequin Bundle Amazon Useful resource Names (ARNs), for adapters produced by SageMaker fine-tuning, or as S3 URIs, for adapters from exterior pipelines equivalent to Hugging Face PEFT. SageMaker AI benchmarks combos of adapter depend and concurrency on actual GPU infrastructure and returns ranked outcomes on your workload. The output lists each codecs for each adapter. The really useful deployment makes use of SageMaker Inference Parts, with one element for the bottom mannequin and one for every adapter, so you possibly can add, replace, or take away an adapter with out disrupting the bottom mannequin or the opposite adapters.
SLA-based concurrency search (September 2026)
Weblog: Proper-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI
Proper-sizing an endpoint means discovering the very best concurrency it could maintain with out breaching latency or error targets. Till now, that meant choosing concurrency ranges by hand, working a sweep, checking the outcomes in opposition to the service degree settlement (SLA), and repeating with a narrower vary at any time when the mannequin or occasion modified.
Inference suggestions now embody a search that finds that boundary immediately. Set search_recipe to max-concurrency-under-sla and provide targets for any mixture of most TTFT, most time per output token, most end-to-end latency, and most error price. The benchmarking engine (AIPerf) begins from a broad vary, narrows it primarily based on which probes cross or fail, and converges in fewer iterations than a linear sweep. The consequence studies the most important passing concurrency (feasible_max) or, if no degree qualifies, the smallest failing one (infeasible_min).
The search makes use of the identical CreateAIWorkloadConfig and CreateAIBenchmarkJob APIs as concurrency sweeps. Non-compulsory parameters (concurrency_min, concurrency_max, search_max_iterations, and search_initial_points) certain the price and pace up convergence.
Agent ability for optimized generative AI inference (October 2026)
Weblog: New agent ability: Amazon SageMaker optimized generative AI inference on your coding agent
Engineers more and more ask their coding agent which occasion and container to make use of. With out grounded information and entry to actual benchmarks, the agent can solely guess, and a fallacious occasion alternative reveals up later as missed latency targets or wasted spend.
The aws-ai-ml ability, out there by means of the Agent Toolkit for AWS, offers coding brokers equivalent to Kiro built-in SageMaker AI inference experience. Engineers describe what they want in plain language, and the agent can do the next:
- Discover the suitable occasion. Ranked deployment choices with measured throughput, latency, and TTFT for a mannequin in Amazon S3, SageMaker JumpStart, or the Hugging Face Hub.
- Benchmark a dwell endpoint. A report from actual load masking throughput, P50 and P99 latency, inter-token latency, and supported concurrency.
- Evaluate runs. The proportion change on each metric, which reveals whether or not a configuration change helped.
Solutions are grounded in measured efficiency and delivered as SageMaker Python SDK v3 code that engineers overview and run below their very own credentials. Setup takes about 10 minutes. The ability works with any agent that helps the Mannequin Context Protocol (MCP) and in SageMaker AI Studio JupyterLab areas.
For the help crew, one advice job that features all 4 adapters, adopted by a concurrency search in opposition to the five hundred millisecond TTFT goal, replaces the handbook sweeps they ran in April. The engineer who owns it could begin the job from a pocket book or ask a coding agent to run it.
Match extra fashions onto every GPU
The help assistant doesn’t run alone. An embedding mannequin, a reranker, and a small classifier that routes tickets dwell in the identical account, and every makes use of a fraction of a GPU. On a typical endpoint, every mannequin copy nonetheless held a complete accelerator, so the crew ran out of placement slots whereas GPU reminiscence sat unused.
Inference Parts within the SageMaker AI console (October 2026)
With Inference Parts, you possibly can host a number of fashions on one endpoint and dimension and scale every mannequin individually. Till now, they had been out there solely by means of the SDK and the AWS Command Line Interface (AWS CLI). Each create, resize, and scaling coverage needed to be scripted, and there was no single view of what every mannequin was utilizing.
Within the SageMaker AI console, now you can create, view, replace, and delete inference elements:
- Create and edit. Create an endpoint with inference elements, add or delete elements later, and alter every element’s accelerators, CPU, reminiscence, and duplicate depend.
- Scale every mannequin. Set an computerized scaling coverage for every element, right down to zero copies, so an idle mannequin stops holding accelerators.
- Monitor. See alarms and per-component metrics on the identical web page.
The console calls the identical SageMaker AI APIs because the SDK and CLI, so groups can begin on the console and automate later, or handle scripted elements visually.
GPU partitioning with NVIDIA Multi-Occasion GPU (October 2026)
Weblog: SageMaker Inference now helps Multi-Occasion GPU to maximise GPU utilization for mannequin endpoints
GPU partitioning makes use of NVIDIA Multi-Occasion GPU (MIG) expertise to divide one bodily GPU into as many as seven partitions, relying on the accelerator. Every partition has devoted GPU compute and reminiscence, with full {hardware} isolation. You outline the partition structure while you create an endpoint for inference elements and select a partition sort for every element. Functions hold invoking fashions by inference element title, so consumer code doesn’t change.
MIG isolates GPU compute and reminiscence between partitions. Fashions on the identical occasion nonetheless share host CPU, host reminiscence, networking, storage, and endpoint routing. You pay the usual price for the complete instance-hour, so the saving comes from inserting extra fashions on every occasion, not from a decrease hourly value.
Determine 2: Three small fashions on three GPUs, in contrast with the identical fashions in partitions of 1 GPU
For the help crew, the embedding mannequin, the reranker, and the classifier now run in separate partitions of 1 GPU on a single endpoint, and the crew manages each element, together with the assistant and its adapters, from the console.
Serve fashions bigger than one occasion
The crew’s subsequent mannequin is a reasoning mannequin whose weights alone exceed the GPU reminiscence of any single occasion. They’ve additionally seen that lengthy, retrieval-heavy prompts sluggish token era for each different person on the endpoint. For a mannequin this dimension, they transfer the workload to HyperPod Inference, which provides them devoted clusters and Kubernetes-native management. The help assistant stays on managed endpoints.
Managed NVIDIA Dynamo on HyperPod Inference (November 2026)
Weblog: Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
Half 1 lined two constructing blocks for giant fashions on HyperPod: disaggregated prefill and decode in Inference Operator v3.2, and KV-aware routing as a part of the managed tiered KV cache. This launch provides multi-node serving and modifications who operates the stack. NVIDIA Dynamo is an open supply inference framework that gives multi-node sharded serving, disaggregated prefill and decode, and KV-aware routing. Groups might already run Dynamo on their very own Kubernetes clusters, however they needed to construct and function every thing round it: inserting all of the nodes of a reproduction collectively, wiring Elastic Cloth Adapter (EFA) networking so prefill can hand its KV cache to decode at {hardware} pace, sustaining serving photographs, and recovering when one failed node takes down a multi-node reproduction.
With this launch, Dynamo runs as a managed serving engine on HyperPod. Dynamo makes the serving selections: how the mannequin is sharded, how prefill and decode are break up, and the place every request goes. HyperPod offers what these selections rely upon:
- Capability and networking. GPU capability and EFA networking, provisioned and wired mechanically.
- Node restoration. Node well being monitoring and alternative, with multi-node employees rescheduled mechanically when a node fails.
- Endpoint operations. Autoscaling, endpoint registration, TLS, and observability.
- One set up and improve path. The Inference Operator add-on installs Dynamo, the serving photographs, and the Grove and KAI schedulers as one unit and upgrades them collectively.
The capabilities Dynamo brings to HyperPod are:
- Multi-node (sharded) serving. One copy of the mannequin spans a number of nodes, so you possibly can serve fashions that don’t match on any single occasion.
- Disaggregated prefill and decode. Immediate processing and token era run on separate GPU swimming pools, every on an occasion sort suited to its work, so an extended immediate now not slows responses for different customers.
- KV-aware routing. Every request goes to the employee that already holds its KV cache, which reduces TTFT.
There are 3 ways to begin. Groups working Dynamo on self-managed Kubernetes can deliver their present DynamoGraphDeployment (DGD) manifests largely unchanged. Groups already on HyperPod Inference can replace the add-on and add a brief dynamoSpec block to an present InferenceEndpointConfig, which HyperPod converts right into a DGD, whereas present endpoints hold serving. Groups new to each can create a cluster with the add-on and Dynamo enabled from the console. In every case, shoppers name a typical OpenAI-compatible endpoint, so consumer code doesn’t change.
Managed Dynamo helps vLLM, SGLang, and TensorRT-LLM, and is out there within the Areas the place Amazon SageMaker HyperPod is out there.
For the help crew, the reasoning mannequin runs as one sharded reproduction throughout a number of nodes, with separate prefill and decode swimming pools. When a node fails, HyperPod replaces it and reschedules the reproduction and not using a web page to the on-call engineer.
Ship every request to a GPU that may serve it
The help assistant now serves clients in North America and Europe, so the crew runs endpoints and HyperPod clusters in two Areas. Through the US morning peak, requests queue within the US Area whereas GPUs the crew already pays for sit idle in Europe. Some European visitors should additionally keep in Europe.
Routing decides whether or not capability a crew owns does helpful work. It occurs at 4 ranges, and the launches beneath cowl one degree every: which occasion serves a request inside an endpoint, which pod inside a HyperPod cluster, which cluster throughout a fleet, and which Regional endpoint throughout Areas and accounts.
Determine 4: Routing throughout Areas and inside every endpoint or cluster, on each deployment paths
KV-cache-aware routing for endpoints (November 2026)
Weblog: Amazon SageMaker AI real-time endpoints now help KV cache conscious routing
Functions with lengthy prompts or multi-turn conversations repeat a big prefix throughout requests. A mannequin server can reuse the matching KV cache, however provided that the following request reaches the occasion that holds it. Prefix-aware routing, lined partly 1, fingerprints the beginning of every immediate and sends matching prompts to the identical occasion. As a result of it really works from request historical past, its view of every occasion’s cache turns into much less correct because the mannequin server provides and evicts entries.
KV-cache-aware routing makes use of the cache state the mannequin container studies. The container publishes cache creation and eviction occasions to a central cache index, and SageMaker AI routes every request to an eligible occasion that holds the longest matching cached prefix. This raises cache reuse for long-context prompts, multi-turn conversations, and workloads that share dynamic immediate prefixes.
It really works with single-model endpoints and endpoints that use inference elements, together with LoRA adapters. It helps InvokeEndpoint, InvokeEndpointWithResponseStream, and the OpenAI-compatible API for Completions and Chat Completions payloads. You flip it on within the endpoint configuration. The primary launch requires an AWS-verified mannequin container that implements the cache-event, tokenization, metrics, and OpenAI-compatible interfaces. You may observe KV cache hit price in detailed inference observability.
Which one to make use of. Prefix-aware routing works with any container and desires no container modifications. KV-cache-aware routing wants a verified container and tracks the precise cache, which helps most when prefixes differ or caches flip over shortly.
Inference Gateway for HyperPod (September 2026)
Weblog: Introducing Amazon SageMaker HyperPod Inference Gateway
Inside a HyperPod cluster, capability and demand not often line up. One pod may be partway by means of a long-context request with its KV cache practically full, whereas the following pod sits idle or already holds the immediate prefix or LoRA adapter the following request wants. With out visibility into that GPU state, new requests queue behind busy pods, and groups add GPUs to guard latency as a substitute of utilizing those they’ve. Every mannequin additionally had its personal endpoint, so shoppers needed to observe one endpoint per mannequin.
The Inference Gateway is a GPU-aware router that installs with the HyperPod Inference Operator add-on and offers every cluster one OpenAI-compatible endpoint for all of its fashions. For every request, it reads dwell indicators from each mannequin pod and picks the most effective one, with weights you possibly can tune per workload:
- KV cache utilization. Avoids pods whose cache is sort of full.
- Queue depth and working requests. Avoids pods with a backlog and balances energetic work.
- Prefix cache hit price. Prefers pods more likely to maintain the immediate prefix.
- LoRA adapter residency. Prefers pods that already maintain the requested adapter, which avoids swap latency.
In contrast with round-robin routing, it diminished P99 TTFT by as much as 97% on blended GPU generations and by as much as 98% below bursty visitors. It’s out there in all Areas the place SageMaker HyperPod is out there.
Cross-Area Inference Routing in Amazon SageMaker AI
On managed endpoints, the identical thought takes the type of the Cross-Area Inference Routing, managed routing that sends real-time and asynchronous requests throughout Regional endpoints a crew has already deployed, in the identical account or in numerous accounts. You group endpoints that serve the identical mannequin below a routing goal, and the appliance invokes the goal as a substitute of a selected Regional endpoint. The router doesn’t add capability, replicate fashions, or deploy endpoints, and the prevailing invocation path retains working.
The router replaces the failover logic groups construct themselves at this time:
- One invocation path. SageMaker tries the native endpoint first and strikes visitors to a different registered endpoint when the native one can’t serve a request.
- One routing coverage. The coverage combines endpoint metrics with service-side capability indicators, together with inadequate capability errors that clients can’t observe on their very own.
- Personal connectivity. Cross-Area visitors runs over AWS PrivateLink with a managed cross-account authorization mannequin, so routing to a different Area or account doesn’t want web or NAT gateways.
- Overflow and residency guardrails. The endpoint proprietor can cap how a lot cross-Area visitors a fallback endpoint accepts, and the appliance can prohibit which Areas could serve a request.
Hold endpoints tuned after launch
Two months after the transfer, the crew provides retrieval to the help assistant, and the common immediate size roughly doubles. TTFT is now greater than it was at launch, however nothing has failed, so each alarm stays inexperienced. When the crew finds a greater configuration, they might want to roll it out on giant GPU cases which can be arduous to get extra of.
Steady efficiency monitoring (November 2026)
Amazon SageMaker AI now repeatedly displays your generative AI inference endpoints and tells you when a change might enhance efficiency or decrease price. Monitoring is on by default for SageMaker AI inference endpoints, with no setup required, and you may decide out of endpoints. SageMaker AI seems solely at operational indicators: latency, GPU utilization, and the endpoint’s serving configuration. It doesn’t learn your requests or responses. When a discovering clears its confidence threshold, a notification seems on the AWS Administration Console, and the identical message arrives by electronic mail for groups that work in notebooks and the CLI. The really useful motion opens the SageMaker AI agentic expertise in SageMaker AI Studio with the discovering already stuffed in, so the evaluation, the comparability, and the deployment change occur in a single dialog. Nothing deploys till you apply it, and you may flip monitoring off at any time.
There are three forms of findings:
- Latency drift after a visitors change. When a change equivalent to added retrieval lengthens prompts, TTFT rises. The discovering quantifies the regression week over week and recommends re-running inference suggestions in opposition to current visitors to discover a configuration that brings TTFT again down.
- A configuration hole current since deployment. The endpoint was by no means effectively tuned for its workload, and the discovering factors to the change that might shut the hole.
- Idle GPU capability. When capability stays unused week after week, the discovering flags an opportunity to right-size the endpoint and scale back price.
In-place rolling mannequin updates (November 2026)
An ordinary rolling replace replaces an endpoint one batch at a time, and each batch runs on newly provisioned cases. On a big GPU fleet, meaning discovering new capability at each step, and an replace can fail partway by means of when the following batch of cases shouldn’t be out there. Every launched occasion should additionally end cleanup earlier than it may be used once more, which slows the rollout.
SageMaker AI now updates fashions in place. It provisions one small batch of extra cases at first, shifts visitors to that batch, after which updates the cases it frees, reusing them to roll the change throughout the remainder of the fleet. The replace nonetheless rolls out in phases, however it wants new capability solely as soon as. For a fleet of 20 cases with a batch dimension of two, the replace acquires 2 new cases as a substitute of 20. Prospects don’t want to vary something. As with container caching partly 1, the advance applies mechanically to rolling mannequin updates.
Determine 6: A rolling replace acquires new cases at each step. An in-place rolling replace acquires one batch at first and reuses the remainder of the fleet
For the help crew, a Drift Watch discovering quantifies the TTFT regression and opens a brand new advice job in opposition to current visitors. They roll out the up to date mannequin with one batch of additional cases, as a substitute of discovering new GPU capability for each occasion within the fleet.
Abstract: 13 launches, persevering with from half 1
The numbering continues from the 13 launches partly 1.
14. Inference suggestions in SageMaker AI Studio (July 2026). A guided, low-code workflow for ranked, benchmarked configurations, with deployment from Studio.
15. Inference suggestions within the SageMaker Python SDK v3 (August 2026). Benchmark, advocate, and deploy from a pocket book or pipeline, with parallel and resumable jobs.
16. LoRA adapter help in inference suggestions (September 2026). Benchmarks adapter depend in opposition to concurrency and deploys adapters as inference elements.
17. SLA-based concurrency search (September 2026). Finds the very best concurrency that meets TTFT, latency, and error-rate targets.
18. Agent ability for optimized generative AI inference (October 2026). Occasion choice, benchmarking, and run comparability from MCP-compatible coding brokers.
19. Inference Parts within the SageMaker AI console (October 2026). Create, scale, and monitor every mannequin on a multi-model endpoint with out scripts.
20. GPU partitioning with NVIDIA MIG (October 2026). As much as seven remoted partitions per GPU, assigned per inference element.
21. Managed NVIDIA Dynamo on HyperPod Inference (November 2026). Multi-node and disaggregated serving, with HyperPod working the platform beneath.
22. KV-cache-aware routing (November 2026). Routes every request to the occasion that holds the longest cached prefix, primarily based on cache occasions from the container.
23. Inference Gateway for HyperPod (September 2026). One endpoint per cluster with GPU-aware routing, and as much as 97% decrease P99 TTFT than round-robin.
24. Cross-Area Inference Router. Routes throughout Regional and cross-account endpoints over PrivateLink, with overflow and residency guardrails.
25. Steady efficiency monitoring. Detects latency drift, configuration gaps, and idle capability, with a guided repair in SageMaker AI Studio.
26. In-place rolling mannequin updates (November 2026). Rolls out mannequin updates with one batch of extra cases at first, then reuses present cases throughout the fleet.
By the top of the yr, the help crew serves 4 product strains from one base mannequin, runs its small fashions on shared GPUs, serves a reasoning mannequin that no single occasion can maintain, and routes visitors throughout two Areas with residency controls. It didn’t must construct its personal benchmarking harness, failover logic, or drift monitoring to get there.
Half 1 argued that benefit in AI inference comes from working an environment friendly inference stack. These 13 launches apply that argument to the months after launch day. The AWS Expertise-Based mostly Acceleration program may help your crew put them to work by benchmarking your workloads and configuring your inference stack in opposition to your SLAs and value targets.
What’s subsequent
On managed endpoints, we plan to increase in-place rolling updates in early 2027 with scale-in-first updates, which take away the necessity for any extra cases throughout a mannequin replace.
Documentation and getting-started guides for each functionality on this submit are within the Amazon SageMaker AI Developer Information. Begin with the part that matches your present downside. In case your visitors has modified because you deployed, re-run inference suggestions. If you’re quick on placement slots, transfer small fashions onto partitioned GPUs. If requests queue whereas capability you personal sits idle, register your clusters and endpoints for cross-Area routing.
References
In regards to the authors




