NVIDIA Nemotron 3.5 Lightning now accessible in Amazon SageMaker JumpStart

NVIDIA Nemotron 3.5 Lightning is designed for the quick, specialised mannequin execution required by high-volume agentic workloads. With NVIDIA Nemotron 3.5 Lightning on Amazon SageMaker JumpStart, you’ll be able to entry an open mannequin designed for high-volume agentic workloads.

With this launch, you’ll be able to deploy Nemotron 3.5 Lightning from Amazon SageMaker JumpStart with out configuring the serving infrastructure your self. NVIDIA describes Lightning because the quickest open mannequin in its class for powering always-on brokers. It delivers as much as 4x greater throughput and as much as 30% quicker activity completion on high-volume agentic workloads. At 30B whole parameters with solely 3B energetic, it may possibly run on a single supported GPU. Repetitive, specialised steps in agent workflows can due to this fact run with out frontier-scale infrastructure. On this publish, we present you the right way to deploy Nemotron 3.5 Lightning from SageMaker JumpStart.

Overview of NVIDIA Nemotron 3.5 Lightning

Nemotron 3.5 Lightning is a publicly accessible basis mannequin distilled from NVIDIA’s frontier Nemotron 3 Extremely and developed with the Nemotron Coalition. It makes use of a hybrid Combination-of-Consultants (MoE) structure and is skilled particularly for agentic instrument use throughout in style agent harnesses. It’s skilled on open datasets and launched as an open mannequin, so you’ll be able to customise it, personal the ensuing weights, and deploy it wherever your brokers run.

The next desk summarizes the important thing specs and efficiency traits of NVIDIA Nemotron 3.5 Lightning.

Not each agent step wants a frontier mannequin

All the time-on brokers work repeatedly: they collect context, observe their surroundings, cause over what they know, and act. Many of those steps can contain mannequin calls, however they don’t all require the identical stage of functionality. Planning a multi-stage workflow or orchestrating sub-agents can demand frontier-level reasoning. Classifying an alert, extracting fields from a kind, or checking a report in opposition to a coverage can usually be dealt with by a smaller, specialised mannequin. These duties can account for a big share of name quantity.

In lots of instances, working all model-backed steps by way of a single massive mannequin can add frontier-model value and latency to work {that a} smaller, specialised mannequin can deal with. A system-of-models method can as a substitute route every step to a mannequin suited to the duty.

Nemotron 3.5 Lightning is constructed for the high-volume finish of that system. Its MoE structure prompts 3B of 30B parameters per ahead move, serving to preserve excessive throughput throughout lengthy, multi-turn classes. DFlash speculative decoding can additional cut back per-token latency. The 1M-token context window permits an agent to hold accrued state throughout a long-running session with out repeated re-grounding.

If NVIDIA NeMo Switchyard is a part of your stack, it may possibly route particular person workflow steps throughout your chosen mannequin pool. Lightning could be chosen for high-volume specialised steps the place its velocity and domain-specific accuracy are match.

Accuracy throughout reasoning and agentic benchmarks

Throughout the printed evaluations, NVFP4 stays near BF16 on many duties, as summarized within the following desk. NVIDIA studies that the analysis recipes and instructions used to supply these outcomes are printed in NeMo Fitness center. Accuracy outcomes have been measured by NVIDIA underneath a constant harness and will differ from distributors’ self-reported numbers.

The next desk compares the BF16 and NVFP4 variants of Nemotron 3.5 Lightning throughout key reasoning and agentic benchmarks.

Customizing for area accuracy

Organizations can post-train the mannequin with NVIDIA NeMo for domain-specific instruments, workflows, and insurance policies, then deploy the ensuing mannequin of their chosen surroundings. The SageMaker JumpStart mannequin card for this launch doesn’t expose JumpStart customization.

Enterprise use instances

Lightning is constructed for the specialised, high-frequency work inside agent workflows:

• Private brokers: Lengthy-running assistants dealing with e-mail, calendar, tasks, and bookings, with the choice to run regionally for contextual information.
• Monetary providers: Extracting information from paperwork, checking coverage guidelines, monitoring danger alerts, and making ready structured summaries.
• Cyber safety operations: Enriching alerts, classifying incidents, querying logs, validating controls, correlating indicators, and making ready findings for analysts.
• Telecom: Triaging community alarms, optimizing community configurations, and answering billing questions.
• Retail: Enriching product catalogs, resolving stock and success exceptions, aiding product discovery, and dealing with order, return, and loyalty questions.

Getting began with SageMaker JumpStart

You’ll be able to deploy Nemotron 3.5 Lightning by way of Amazon SageMaker JumpStart with out manually configuring the serving framework.

Conditions

Earlier than you start, be sure to have:

• An AWS account.
• Appropriately scoped permissions for SageMaker JumpStart.
• Adequate service quota for GPU situations (for instance, ml.g6e.12xlarge, ml.p4d.24xlarge, or ml.p5.48xlarge).

Necessary: Deploying this mannequin creates a SageMaker AI endpoint that incurs costs whereas working. For particulars, see Amazon SageMaker AI pricing. Delete your endpoint when completed to keep away from ongoing costs.

Deploy utilizing SageMaker Studio

1. Open Amazon SageMaker Studio.
2. Within the navigation pane, select SageMaker JumpStart.
3. Seek for Nemotron 3.5 Lightning.
   

   Determine 1: Trying to find Nemotron 3.5 Lightning in SageMaker JumpStart

4. Choose the mannequin card. The NVFP4 mannequin ID is huggingface-reasoning-nemotron-3-5-lightning-30b-a3b-nvfp4. For BF16, use huggingface-reasoning-nemotron-3-5-lightning-30b-a3b-bf16.
   

   Determine 2: The Nemotron 3.5 Lightning mannequin card in SageMaker JumpStart

   

   Determine 3: Mannequin card particulars for Nemotron 3.5 Lightning

5. Select Deploy.
6. Choose your occasion sort (for instance, ml.g6e.24xlarge).
   

   Determine 4: Deciding on an occasion sort for deployment

7. Evaluation the deployment settings. The defaults are ample for many use instances.
8. Select Deploy to create the endpoint.
9. Look ahead to the endpoint standing to point out InService earlier than working inference.
   

   Determine 5: Endpoint standing displaying InService

Deploy from Hugging Face

You too can deploy NVIDIA Nemotron 3.5 Lightning to Amazon SageMaker AI from its Hugging Face mannequin web page. On the Hugging Face mannequin web page, select Deploy, choose Amazon SageMaker AI, after which select Deploy on SageMaker AI. This opens the SageMaker AI deployment workflow, the place you’ll be able to configure and deploy the mannequin.

Determine 6: Deploying from the Hugging Face mannequin web page

Deploy utilizing the SageMaker Python SDK

With SageMaker JumpStart, you’ll be able to entry the NVFP4 and BF16 variants. The next instance makes use of the NVFP4 mannequin ID huggingface-reasoning-nemotron-3-5-lightning-30b-a3b-nvfp4. For BF16, use huggingface-reasoning-nemotron-3-5-lightning-30b-a3b-bf16.

from sagemaker.jumpstart.mannequin import JumpStartModel

model_id = "huggingface-reasoning-nemotron-3-5-lightning-30b-a3b-nvfp4"
model_version = "*"

mannequin = JumpStartModel(model_id=model_id, model_version=model_version)
predictor = mannequin.deploy()

Clear up

To keep away from pointless costs, delete the SageMaker AI endpoint if you end up executed:

predictor.delete_endpoint()

Conclusion

NVIDIA Nemotron 3.5 Lightning brings quick, specialised agent execution to Amazon SageMaker JumpStart. NVIDIA studies as much as 4x greater throughput and as much as 30% quicker activity completion for high-volume specialised work. Its hybrid MoE structure, 3B energetic parameters, DFlash speculative decoding, and 1M-token context are designed for high-volume agent workflows. As a result of the mannequin is open and designed for personalisation, you’ll be able to post-train it in your personal instruments and insurance policies and retain management of the ensuing mannequin.

Nemotron 3.5 Lightning can help specialised agent workloads throughout private assistants, monetary providers, safety operations, telecom, and retail. You’ll be able to deploy it in the present day from SageMaker JumpStart.

Get began by looking for Nemotron 3.5 Lightning in Amazon SageMaker JumpStart. For deployment steerage, see JumpStart basis mannequin utilization within the Amazon SageMaker AI Developer Information.

Concerning the authors