Automationscribe.com
  • Home
  • AI Scribe
  • AI Tools
  • Artificial Intelligence
  • Contact Us
No Result
View All Result
Automation Scribe
  • Home
  • AI Scribe
  • AI Tools
  • Artificial Intelligence
  • Contact Us
No Result
View All Result
Automationscribe.com
No Result
View All Result

Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

admin by admin
November 18, 2026
in Artificial Intelligence
0
Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
399
SHARES
2.3k
VIEWS
Share on FacebookShare on Twitter


Serving massive language fashions (LLMs) effectively requires adequate GPU reminiscence and cautious coordination of immediate processing and token era. GLM-5.3’s FP8 weights, for instance, exceed the 640 GB of GPU reminiscence on a single ml.p5.48xlarge occasion, earlier than accounting for KV cache and runtime overhead. Lengthy prompts may sluggish token era when each share the identical GPUs.

Amazon SageMaker HyperPod Inference now helps NVIDIA Dynamo, combining Dynamo’s distributed inference capabilities with the infrastructure administration of HyperPod to deal with these challenges. Dynamo is an open supply distributed inference framework that helps multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines resembling vLLM, SGLang, and TensorRT-LLM. HyperPod helps these deployments with high-performance Elastic Cloth Adapter (EFA) networking, coordinated GPU scheduling, and node well being monitoring and restoration. On this publish, we illustrate multi-node and disaggregated serving by means of a deployment instance, share benchmark outcomes, and description finest practices for optimizing inference on HyperPod.

How the mixing works

The HyperPod Inference add-on installs the Dynamo customized useful resource definitions and these serving and scheduling elements:

  • HyperPod Inference Operator manages endpoint deployment by means of an InferenceEndpointConfig (IEC) customized useful resource. Its dynamoSpec part provides the Dynamo configuration.
  • Dynamo operator manages assets outlined in a DynamoGraphDeployment (DGD), together with the frontend, mannequin staff, duplicate counts, and useful resource necessities.
  • Grove coordinates associated teams of pods, together with multi-node staff, and their scaling relationships.
  • Kubernetes AI (KAI) Scheduler locations pods on GPU assets, utilizing gang scheduling to schedule the pods required for a multi-node employee collectively.

AWS Deep Studying Containers (DLCs) present the Dynamo runtime, whereas the add-on manages built-in element variations. To undertake a supported Dynamo launch, replace the add-on.

Deployment structure

The next diagram reveals the deployment paths, serving elements, and infrastructure administration for disaggregated inference on SageMaker HyperPod. The Dynamo frontend routes requests to prefill staff, which course of prompts, and decode staff, which generate tokens. These separate swimming pools enable unbiased scaling and tuning. Each swimming pools load the total mannequin weights, and KV cache transfers between them over EFA.

Dynamo additionally helps aggregated serving, the place prefill and decode run in the identical employee. In both topology, staff can run on one occasion or span a number of cases utilizing tensor and pipeline parallelism. Dynamo’s KV-aware router considers reusable prefix KV cache and lively employee load in each topologies. Prefix reuse can scale back immediate recomputation and enhance time to first token, whereas load consciousness helps keep away from overloading staff. For extra particulars on the Dynamo structure, see the Dynamo Information Base overview.

HyperPod supplies node, GPU, and EFA well being monitoring and manages node alternative, whereas the orchestration elements recreate affected staff. Deployments configured by means of InferenceEndpointConfig with dynamoSpec assist invocation by means of SageMaker endpoints.

Architecture diagram of NVIDIA Dynamo on SageMaker HyperPod with the frontend routing to prefill and decode workers over EFA

Determine 1: Deployment paths, serving elements, and infrastructure administration for NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

Two methods to configure your deployment

Whether or not you might be new to HyperPod, new to Dynamo, or each, the add-on affords two deployment paths to suit your workflow:

Convey-your-own DynamoGraphDeployment (DGD) lets current Dynamo customers proceed utilizing upstream documentation, recipes, and manifests, with the Dynamo operator managing the deployment. To carry an current DGD to HyperPod, use a supported container picture and configure storage, networking, and pod placement on your cluster.

InferenceEndpointConfig with dynamoSpec lets current HyperPod Inference customers deploy a brand new Dynamo endpoint utilizing the identical configuration format and OpenAI-compatible interface they already know. For the inference engine, you possibly can select vLLM, SGLang, or TensorRT-LLM, with vLLM utilized by default. The operator additionally provides defaults for KV-aware routing and duplicate counts.

Each paths assist single-instance and multi-node deployments. In both case, HyperPod manages node well being and alternative. The desk summarizes the tasks for every path.

Convey-your-own DGD InferenceEndpointConfig with dynamoSpec
You write Full DGD manifest IEC containing dynamoSpec
Greatest for Groups wanting direct management of the serving graph or reuse of current DGDs Groups wanting managed endpoint setup by means of IEC
Mannequin weights You configure downloading or mounting the weights The operator levels weights from Amazon Easy Storage Service (Amazon S3) or Amazon FSx
Endpoint, TLS, registration You configure and handle these assets Managed by means of the HyperPod Inference Operator
Autoscaling You configure and handle scaling Kubernetes Occasion-driven Autoscaling (KEDA) by means of autoScalingSpec for aggregated and disaggregated deployments

Deployment instance: GLM-5.3

The next excerpts use the InferenceEndpointConfig (IEC) deployment path with vLLM on ml.p5.48xlarge cases. Dynamo-specific settings belong underneath spec.dynamoSpec. Mannequin supply and endpoint settings are included within the full manifest.

Multi-node aggregated serving

This configuration runs one model-worker duplicate throughout two cases (16 GPUs), with two frontend replicas for request dealing with. Setting dynamoSpec.frontend.routingMode to kv-aware activates KV-aware routing.

spec:
  replicas: 1
  dynamoSpec:
    modelServer: vllm
    frontend:
      replicas: 2
      routingMode: kv-aware
    aggregation:
      nodesPerReplica: 2
      tensorParallelism: 16

Right here, spec.replicas controls the variety of model-worker replicas, whereas frontend.replicas controls the variety of frontend replicas.

Multi-node disaggregated serving

Substitute aggregation underneath spec.dynamoSpec with the next disaggregation block, protecting the previous frontend configuration. Duplicate counts are set individually for prefill and decode. This instance runs one prefill duplicate and two decode replicas, every spanning two cases, for a complete of six cases.

disaggregation:
  prefill:
    nodesPerReplica: 2
    tensorParallelism: 16
    replicas: 1
  decode:
    nodesPerReplica: 2
    tensorParallelism: 16
    replicas: 2

Apply the deployment

Apply the whole manifest with kubectl apply. The HyperPod Inference Operator makes the configured mannequin weights out there to the employees and generates the DGD. Grove and KAI Scheduler coordinate placement in order that the pods required for every multi-node employee are scheduled collectively.

After the SageMaker endpoint is prepared and configured for chat completions, shoppers can ship an OpenAI-compatible request payload by means of the SageMaker Runtime API:

aws sagemaker-runtime invoke-endpoint 
--endpoint-name glm53 
--content-type utility/json 
--body '{"mannequin":"zai-org/GLM-5.3","messages":[{"role":"user","content":"Hello"}],"max_tokens":128}' 
--cli-binary-format raw-in-base64-out 
out.json

Benchmarks

We benchmarked Dynamo on SageMaker HyperPod throughout two fashions to check serving configurations.

Disaggregated prefill and decode for Qwen3-235B

We used AIPerf to check aggregated and disaggregated serving for Qwen3-235B-A22B-FP8 utilizing Dynamo with TensorRT-LLM on two p5en.48xlarge cases (16 H200 GPUs). Requests contained 4,000 enter tokens and 200 output tokens. We retained measured factors that met two targets: P90 time to first token under one second, and median era velocity of not less than 50 tokens per second per consumer. Amongst these factors, disaggregated serving achieved 2.05× the best aggregated throughput: 207 versus 101 output tokens per second per GPU, at concurrency 80 and 32, respectively.

Throughput comparison for Qwen3-235B: disaggregated serving at 207 versus aggregated at 101 output tokens per second per GPU

Determine 2: Aggregated in comparison with disaggregated serving throughput for Qwen3-235B-A22B-FP8

Multi-node aggregated serving for Qwen3.5-397B

Dynamo helps multi-node serving in each aggregated and disaggregated modes for fashions that exceed a single occasion’s GPU reminiscence. To judge a employee spanning nodes, we used AIPerf to benchmark multi-node aggregated serving for Qwen3.5-397B-A17B-FP8 with Dynamo and vLLM, utilizing TP4 × PP2 and eight lively H200 GPUs. This mannequin matches on one occasion, so in observe you wouldn’t cut up it. It serves right here as a managed baseline. We in contrast eight GPUs on one p5en.48xlarge in opposition to 4 GPUs on every of two cases, utilizing the identical mannequin, parallelism, and workload configuration. With an agentic workload that includes excessive prefix reuse at concurrency 32, the two-node configuration delivered 126.4 output tokens per second per GPU, retaining 99.1 % of single-node throughput. This reveals that Dynamo on SageMaker HyperPod can serve a mannequin that wants multiple occasion throughout nodes. The node boundary itself doesn’t develop into a big throughput penalty.

Throughput comparison for Qwen3.5-397B: two-node serving at 126.4 output tokens per second per GPU, 99.1% of single-node throughput

Determine 3: Single-node in comparison with two-node throughput for multi-node aggregated serving of Qwen3.5-397B-A17B-FP8

Greatest practices for optimized inference

NVIDIA Dynamo supplies recipes and instruments that can assist you choose a serving configuration, benchmark efficiency, and tune your deployment on SageMaker HyperPod.

  • Begin with an identical recipe – Select a Dynamo recipe on your mannequin, inference engine, and GPU kind. Account for mannequin weights, KV cache, and runtime reminiscence when sizing staff. For supported configurations, AISimulate may also help examine aggregated and disaggregated serving, select parallelism, and dimension prefill and decode employee swimming pools earlier than testing on GPUs.
  • Confirm communication earlier than benchmarking – For deployments spanning cases, affirm that communication makes use of the supposed EFA transport, together with KV-cache transfers between prefill and decode staff. See the Dynamo EFA information.
  • Benchmark consultant visitors – Replay your visitors with AIPerf, or use a generated agentic hint with consultant immediate lengths, output lengths, and prefix reuse. Sweep concurrency for every configuration, retain factors assembly your SLAs, and select primarily based on price and latency priorities.
  • Tune from measured bottlenecks – Regulate topology and employee allocation earlier than engine settings, altering one variable at a time. Observe the efficiency tuning information, or use Dynamo optimization agent abilities to help with benchmarking and tuning. Validate every candidate in your goal HyperPod deployment.

Conclusion

NVIDIA Dynamo on SageMaker HyperPod Inference simplifies deploying and working massive fashions with multi-node serving, disaggregated prefill and decode, and KV-aware routing. The managed add-on installs the serving elements, and configuration defaults simplify endpoint setup. HyperPod manages mannequin weights, auto-scales clusters primarily based on GPU and inference metrics, displays node well being, and replaces failed nodes. Collectively, these capabilities scale back undifferentiated infrastructure work so groups can give attention to their fashions and functions. Get began with the setup directions and full deployment examples within the linked repository.

Acknowledgements

The authors want to thank Eliuth Triana, Felipe Garcia, Sanjana Checker, Junyuan Yang, Jiahong Liu, Anish Maddipoti, Jacob Kern and Abdullahi Olaoye from NVIDIA for his or her contributions to this publish. We additionally want to thank Can Solar, Ophelia Yang, Aryaman Kothari, David Oh, and Ayush Kumar from AWS for his or her contributions.


In regards to the authors

Vinay Arora

Vinay Arora

Vinay is a Specialist Answer Architect for Generative AI at AWS, the place he collaborates with prospects in designing cutting-edge AI options, leveraging AWS applied sciences and Infrastructure for the previous 6 years. Previous to AWS, Vinay has over twenty years of expertise in finance—together with roles at banks and hedge funds—he has constructed threat fashions, buying and selling techniques, and market information platforms. Vinay holds a grasp’s diploma in laptop science and enterprise administration.

Piyush Daftary

Piyush Daftary

Piyush is a Principal Software program Engineer at AWS, the place he builds inference techniques for giant language fashions on Amazon SageMaker HyperPod. He focuses on optimized routing, disaggregated serving, and autoscaling architectures that assist prospects extract most efficiency from their GPU fleets. His background spans distributed techniques, databases, and search applied sciences. Exterior of labor, he enjoys touring and spending time with household.

Xuan Lu

Xuan Lu

Xuan is a Software program Improvement Engineer at AWS, the place he works on Amazon SageMaker HyperPod Inference to construct and scale large-scale LLM inference techniques. His work focuses on distributed inference, clever request routing, and efficiency optimization, with expertise throughout Kubernetes-based orchestration and LLM serving infrastructure. Exterior of labor, he enjoys touring, exploring nature, and studying science fiction.

Preethi Ravula

Preethi Ravula

Preethi is a Options Structure Supervisor for Generative AI, Information, Storage & Analytics on the ISV workforce at AWS. She leads a workforce that helps software program distributors construct and scale generative AI on AWS. She works on AI technique and inference optimization, and he or she develops frameworks and playbooks that assist prospects run massive language mannequin workloads sooner and extra cost-effectively on AWS Trainium and NVIDIA GPUs. Preethi has spoken at GHC 2025, SJSU, and Ladies in Tech Dublin. She additionally champions AI training by means of an AWS program that teaches college students to construct functions with AWS generative AI companies.

Sai Devulapalli

Sai Devulapalli

Sai has been main the Amazon SageMaker AI world-wide GTM workforce over the previous six years. He works carefully with customer-facing AI Specialist leaders throughout AWS world-wide and engages with C/VP-level stakeholders at prospects to grasp their wants and supply steering in constructing and executing on their firm’s AI platform methods. Sai works with SageMaker AI Service management to assist information service roadmap to higher serve our buyer wants and has been in AI/ML platform house over the previous 16 years, main enterprise growth and product administration groups at Dell EMC and Pivotal and previous to that, main Information Analytics and ML observe for Ericsson in North America.

Aryaman Kothari

Aryaman is a Senior Product Supervisor at Amazon SageMaker AI, the place he leads AI inference on Amazon SageMaker HyperPod, serving to enterprises run AI fashions at scale for manufacturing workloads. A Chicago Sales space graduate, Aryaman beforehand led product and go-to-market for Quantiphi’s Conversational AI enterprise. Exterior work, he enjoys taking part in soccer, making chai, and climbing.

Kshitiz Gupta

Kshitiz Gupta

Kshitiz is a Senior Options Architect at NVIDIA, the place he helps cloud prospects design and optimize GPU-accelerated information and AI techniques. His work spans information processing, vector database, and LLM inference, partnering carefully with AWS and Amazon groups to carry these capabilities into manufacturing. Exterior of labor, he enjoys music, yoga, and climbing.

Marc Karp

Marc Karp

Marc is a Senior Options Architect at NVIDIA, the place he helps prospects deploy and scale AI inference on GPUs. He works throughout the inference stack, from mannequin optimization and distributed serving to the engines that energy them. Exterior of labor, he enjoys touring and exploring new locations.

Wenhan Tan

Wenhan Tan

Wenhan is a options architect at NVIDIA, helping prospects in adopting NVIDIA AI options at large-scale. His work focuses on accelerating deep studying functions and addressing inference and coaching challenges.

Tags: AmazonDynamoHyperPodInferenceManagedNVIDIASageMaker
Previous Post

How ArisGlobal scaled generative AI inference on Amazon Bedrock

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Popular News

  • Speed up edge AI improvement with SiMa.ai Edgematic with a seamless AWS integration

    Speed up edge AI improvement with SiMa.ai Edgematic with a seamless AWS integration

    407 shares
    Share 163 Tweet 102
  • The Journey from Jupyter to Programmer: A Fast-Begin Information

    407 shares
    Share 163 Tweet 102
  • Greatest practices for Amazon SageMaker HyperPod activity governance

    407 shares
    Share 163 Tweet 102
  • Context Engineering — A Complete Fingers-On Tutorial with DSPy

    407 shares
    Share 163 Tweet 102
  • Construct a serverless audio summarization resolution with Amazon Bedrock and Whisper

    406 shares
    Share 162 Tweet 102

About Us

Automation Scribe is your go-to site for easy-to-understand Artificial Intelligence (AI) articles. Discover insights on AI tools, AI Scribe, and more. Stay updated with the latest advancements in AI technology. Dive into the world of automation with simplified explanations and informative content. Visit us today!

Category

  • AI Scribe
  • AI Tools
  • Artificial Intelligence

Recent Posts

  • Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
  • How ArisGlobal scaled generative AI inference on Amazon Bedrock
  • Construct memory-aware AI brokers with Agentic Retrieve on Amazon Bedrock Managed Information Base
  • Home
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Terms & Conditions

© 2024 automationscribe.com. All rights reserved.

No Result
View All Result
  • Home
  • AI Scribe
  • AI Tools
  • Artificial Intelligence
  • Contact Us

© 2024 automationscribe.com. All rights reserved.