Deploying quantized fashions on Amazon SageMaker AI with Unsloth

This put up was co-written with Daniel Han and Michael Han from Unsloth.

Deploying giant basis fashions (FMs) saved at their unique 16-bit floating-point precision (BF16 or FP16) is dear. They want giant GPU situations, driving up serving prices, and slowing down iteration cycles. Quantization addresses this by lowering the numerical precision of a mannequin’s weights (for instance from 16-bit to 4-bit), which shrinks the reminiscence utilization considerably. The disadvantage of quantization is that it could scale back the accuracy of a mannequin, which is the place dynamic quantization turns into compelling. When carried out accurately, dynamic quantization can scale back reminiscence utilization whereas sustaining accuracy. The financial savings in occasion price, storage, and startup time can compound rapidly at scale.

On this put up, you’ll study 4 deployment patterns for taking fashions which have already been quantized with Unsloth and deploying them on AWS infrastructure. The patterns use Amazon Elastic Compute Cloud (Amazon EC2) for direct occasion entry, Amazon SageMaker AI inference endpoints for managed serving, and Amazon Elastic Kubernetes Service (Amazon EKS) or Amazon Elastic Container Service (Amazon ECS) when inference wants to suit into an current container framework. You additionally study operational practices for manufacturing deployments.

What’s Unsloth Dynamic quantization?

As Daniel Han, co-founder of Unsloth, explains:

“The most important drawback of a robust mannequin is it’s very large and also you want 1.5TB to run this mannequin. By utilizing some methods you can also make the mannequin 217GB in measurement. You would possibly assume as a result of it’s 86% smaller, accuracy will degrade by 86%, however that’s not the case, it solely degrades by 14% accuracy. This system we name it dynamic quantization and we present many benchmarks the place you’ll be able to scale back the disk area of the mannequin, by not quantizing all weights all the way down to 4-bit and so forth. however some layers stay in increased 8-bits.”

In sensible phrases, quantization reduces the variety of bits used to retailer every weight. A normal BF16 mannequin makes use of 16 bits per parameter. Quantizing to 4-bit shrinks the dimensions by 75 p.c, although real-world file sizes are barely bigger due to quantization metadata. For an 8-billion parameter mannequin, that takes the reminiscence footprint from roughly 16 GB all the way down to roughly 5 GB. That’s typically the distinction between needing a multi-GPU occasion and becoming comfortably on a single GPU.

Unsloth is a software for fine-tuning and quantizing basis fashions. Unsloth Dynamic is a quantization methodology that goes past uniform compression. Relatively than making use of the identical bit discount to each layer, it really works in three steps:

1. Layer-by-layer evaluation – Unsloth measures how delicate every layer is to precision loss.
2. Dynamic bit allocation – Necessary layers (these the place precision loss causes significant output degradation) are stored at increased precision (for instance, 16-bit), whereas much less delicate layers are quantized aggressively (4-bit or decrease).
3. Precision tuning – The quantization is tuned in order that the mixed output high quality stays as shut as potential to the unique, whereas maintaining disk area utilization as small as potential.

The tip aim of this course of is to make the accuracy variations between the quantized mannequin and the usual mannequin as small as potential, whereas additionally compressing the mannequin measurement by a significant quantity. With the open supply Unsloth package deal, you’ll be able to fine-tune, run, export, and deploy fashions in a single unified workflow.

Whenever you deploy on AWS, quantization issues as a result of it modifications three issues concurrently. First, the occasion choice: a big mannequin that will in any other case require bigger GPUs would possibly develop into sensible on a smaller one, and even on CPU. Second, the startup and storage profile: smaller mannequin recordsdata transfer, retailer, and promote throughout environments sooner. Third, the deployment flexibility: you’ll be able to select a smaller mannequin file for cost-sensitive inference, a higher-fidelity export for quality-sensitive inference, or a merged illustration for higher-throughput GPU serving. That flexibility is what makes Unsloth helpful in an AWS surroundings. With it, you’ll be able to adapt the mannequin to the serving path as an alternative of forcing each deployment into the identical runtime and {hardware} assumptions.

Which mannequin format to make use of

A key precept within the Unsloth deployment workflow is that the output artifact ought to drive the serving design. Infrastructure choice comes second. First, select the artifact and runtime, then place that runtime on the AWS service that matches your working mannequin.

Unsloth helps a number of deployment-oriented output varieties:

• GGUF recordsdata – GGUF is a single-file format that packages mannequin weights, tokenizer, and metadata collectively, making it self-contained and able to load with out extra recordsdata. Use these for light-weight runtimes equivalent to llama.cpp, Ollama, and Unsloth. On AWS, this maps to Amazon EC2 or an Amazon SageMaker AI customized container.
• Merged safetensors weights – (16-bit, 8-bit, FP8 4-bit, NVFP4) may be created by means of Unsloth for higher-throughput engines equivalent to vLLM and SGLang. On AWS, this maps to Amazon SageMaker AI Massive Mannequin Inference (LMI) containers, Amazon EKS, or Amazon ECS.

A sensible deployment map

The next desk maps every artifact kind to its best-fit runtime and AWS goal, so you’ll be able to rapidly establish which deployment sample matches your necessities.

Every deployment sample on this put up follows the identical normal workflow:

1. Tremendous-tune or obtain a mannequin in Unsloth.
2. Export the mannequin file that matches the runtime you need.
3. Validate the runtime regionally or on Amazon EC2.
4. Promote the identical mannequin file and runtime mixture to a managed or environment-native deployment.

That sequencing helps keep away from surprising conduct later, particularly round reminiscence use, immediate formatting, and latency conduct.

Sample 1: GGUF on Amazon EC2 with llama.cpp and Unsloth

Use this sample to validate quantization ranges rapidly with direct occasion entry. For a lot of use instances, Amazon EC2 is a powerful start line since you get most management with minimal abstraction. Benchmark a number of quantization ranges, take a look at immediate formatting, evaluate CPU versus GPU conduct, and perceive the precise reminiscence footprint earlier than you decide to a managed endpoint design.

A typical Unsloth export appears to be like like this:

mannequin.save_pretrained_gguf("gguf_model", tokenizer, quantization_method="q4_k_xl")

This produces a GGUF file prepared for native llama.cpp-based inference. The companion repository makes use of q4_k_xl for its quantized endpoint. Totally different quantization strategies management the trade-off between file measurement and output constancy: use q8_0 for increased output constancy at roughly double the file measurement, or f16 for a full-precision GGUF with no quantization loss. See the Unsloth GGUF documentation for the whole listing of quantization strategies and their traits.

After you could have the GGUF, a minimal llama-server launch requires the next configurations:

llama-server 
  --model /fashions/my-model.gguf 
  --alias my-model 
  --ctx-size 8192 
  --host 0.0.0.0 
  --port 8080

unsloth run 
  --model /fashions/my-model.gguf 
  --alias my-model 
  --ctx-size 8192 
  --host 0.0.0.0 
  --port 8080

At that time, your software can speak to the mannequin by means of an OpenAI-compatible interface:

from openai import OpenAI

consumer = OpenAI(
    base_url="http://:8080/v1",
    api_key="not-required",
)
response = consumer.chat.completions.create(
    mannequin="my-model",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain AWS Graviton in two sentences."},
    ],
    max_tokens=128,
)
print(response.selections[0].message.content material)

⚠️ Necessary: This configuration is appropriate for remoted testing. For manufacturing, prohibit the safety group to identified Classless Inter-Area Routing (CIDR) blocks, bind to a personal interface (for instance, 127.0.0.1), and place the endpoint behind an authenticated API gateway or load balancer.

This path is enticing for 3 causes. It’s clear to debug, it stays near upstream runtime conduct, and it provides you a clear place to check quantization selections earlier than you add container contracts, autoscaling insurance policies, or endpoint governance.

Amazon EC2 can also be essentially the most direct place to guage sensible deployment questions:

• Quantization high quality – Evaluate outputs throughout quantization ranges (for instance, q4_k_m versus q8_0) towards your particular analysis standards. High quality impression varies by mannequin structure and process.
• {Hardware} necessities – If the mannequin’s reminiscence footprint exceeds out there CPU RAM, you want a GPU-backed occasion. Verify peak reminiscence utilization throughout inference, not solely mannequin load. For greatest efficiency, have adequate RAM or VRAM to load the complete mannequin into reminiscence. For instance, if the mannequin is 128 GB disk, goal for greater than 128 GB of obtainable RAM or VRAM.
• Context size trade-offs – Longer context home windows improve each latency and reminiscence consumption. Take a look at with consultant immediate lengths to search out the proper stability in your workload.
• Chat template validation – Confirm that the structured immediate format (which maps roles equivalent to “system” and “consumer” to the token sequences the mannequin expects) produces constant conduct exterior the coaching surroundings.

For analysis, inner instruments, proof-of-concept work, and early manufacturing pilots, Amazon EC2 is usually the proper start line.

Sample 2: GGUF on Amazon SageMaker AI with a customized container

Amazon EC2 is really useful for quick iteration, nevertheless it doesn’t provide you with a managed inference service out of the field. Whenever you desire a manufacturing endpoint with autoscaling, monitoring, AWS Id and Entry Administration (IAM) integration, and a steady API floor, Amazon SageMaker AI inference endpoints develop into the higher match.

For GGUF deployments, essentially the most sensible design is normally a customized inference container that packages the GGUF mannequin file along with llama.cpp or one other light-weight runtime. The container then exposes the Amazon SageMaker AI internet hosting interface, which requires that the container pay attention on port 8080, implement /ping for well being checks, and /invocations for inference requests.

The companion GitHub repository features a full working instance of this sample. It deploys a side-by-side comparability of Unsloth’s dynamically quantized Qwen3-VL-8B-Instruct (Q4_K_XL GGUF served with llama.cpp on ml.g5.xlarge, roughly $1.41/hr) towards the full-precision BF16 variant (served with vLLM on ml.g5.12xlarge, roughly $7.09/hr). Pricing as of June 2026; see the Amazon SageMaker AI pricing web page for present charges. The entrypoint script begins llama-server on an inner port and makes use of nginx as a reverse proxy to fulfill the Amazon SageMaker AI interface:

# Begin llama-server with the GGUF mannequin and imaginative and prescient projector
/app/llama-server 
  --model /fashions/Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf 
  --mmproj /fashions/mmproj-F16.gguf 
  --host 0.0.0.0 
  --port 8081 
  --ctx-size 4096

Nginx then maps /ping to the llama-server well being verify and /invocations to the chat completions endpoint, giving Amazon SageMaker AI a normal interface whereas maintaining the runtime light-weight:

# SageMaker well being verify endpoint
location /ping {
    proxy_pass http://llama_backend/well being;
}

# SageMaker inference endpoint -> llama.cpp chat completions
location /invocations {
    proxy_pass http://llama_backend/v1/chat/completions;
    proxy_read_timeout 120s;
}

A typical structure would possibly seem like this:

1. Retailer the finalized GGUF mannequin file in Amazon Easy Storage Service (Amazon S3).
2. Begin the Amazon SageMaker AI container and place the mannequin underneath /decide/ml/mannequin.
3. Launch llama.cpp or Unsloth towards the native mannequin file.
4. Expose a skinny API layer that interprets Amazon SageMaker AI requests into native inference calls.

The next structure diagram from the companion GitHub repository exhibits how the pattern deploys each a quantized GGUF endpoint (this sample) and a full-precision vLLM endpoint (Sample 3). The circulate works as follows:

Determine: Structure diagram of the companion pattern deployment.

1. Terraform provisions all infrastructure together with Amazon Elastic Container Registry (Amazon ECR), AWS CodeBuild, and Amazon S3 sources.
2. AWS CodeBuild builds the customized container picture and pushes it to Amazon ECR.
3. Amazon SageMaker AI creates two real-time endpoints inside a digital personal cloud (VPC): one for the quantized GGUF mannequin (Sample 2) and one for the full-precision mannequin (Sample 3).
4. Each endpoints pull their respective mannequin recordsdata from Amazon S3 at startup.
5. An Amazon SageMaker Pocket book Occasion gives entry to the comparability pocket book for operating evaluations towards each endpoints.

This sample works properly as a result of it retains the runtime light-weight whereas letting Amazon SageMaker AI deal with the operational considerations that matter in manufacturing, equivalent to endpoint lifecycle and autoscaling. For smaller fashions, this path is particularly compelling on CPU-backed endpoints. For bigger quantized fashions or lower-latency necessities, you’ll be able to transfer the identical design to GPU-backed endpoints.

A trade-off is that llama.cpp is the inference/deployment engine, however not the entire serving system. In Amazon SageMaker AI, you continue to want a container that behaves like an Amazon SageMaker AI endpoint. That wrapper layer doesn’t require important implementation effort, nevertheless it’s an essential a part of the manufacturing design.

Sample 3: Merged weights on Amazon SageMaker AI with GPU-optimized serving engines

GGUF is superb when mannequin file measurement and light-weight serving are the priorities. It’s not at all times the proper alternative while you prioritize throughput and GPU effectivity. Merged weights develop into extra enticing in these situations.

Merged weights consult with a mannequin saved as a single unified set of safetensors weights. That is typically the results of combining a fine-tuned adapter (equivalent to a Low-Rank Adaptation (LoRA), adapter) with the bottom mannequin, nevertheless it additionally applies to base fashions saved instantly in a deployment-ready format. Unsloth helps save paths for merged 16-bit and merged 4-bit outputs, together with LoRA-only exports. These are extra aligned with runtimes equivalent to vLLM and SGLang which are constructed round manufacturing GPU serving patterns equivalent to batching, excessive token throughput, and multi-GPU scaling.

A consultant export appears to be like like this:

mannequin.save_pretrained_merged(
    "finetuned_model",
    tokenizer,
    save_method="merged_16bit",
)

The serving path is:

vllm serve finetuned_model

On AWS, the LMI containers out there for Amazon SageMaker AI are particularly related right here. For the full-precision endpoint within the companion pattern, the LMI container for Amazon SageMaker AI is configured solely by means of surroundings variables on the Terraform mannequin useful resource:

HF_MODEL_ID = "Qwen/Qwen3-VL-8B-Instruct"
OPTION_DTYPE = "bf16"
OPTION_ROLLING_BATCH = "vllm"
OPTION_TENSOR_PARALLEL_DEGREE = "4"
OPTION_MAX_MODEL_LEN = "4096"

No customized container is required. The LMI container downloads the mannequin from Hugging Face at startup and serves it by means of vLLM with tensor parallelism throughout 4 A10G GPUs. Tensor parallelism splits a mannequin throughout a number of GPUs so every GPU handles a portion of the computation in parallel. If the workload calls for steady batching, increased concurrency, or bigger mannequin footprints, a vLLM-style path is usually a greater manufacturing alternative than a light-weight GGUF runtime. The deployment priorities shift from match the mannequin cheaply to serve the mannequin effectively at scale, and this sample addresses that shift instantly.

A helpful rule of thumb: use GGUF and llama.cpp while you need the lightest deployment path and the runtime matches the workload. Use merged weights (the mixed base mannequin and adapter output) with vLLM, SGLang, or an LMI backend when throughput, batching, concurrency, or {hardware} topology develop into the dominant constraints.

Sample 4: Use Amazon EKS or Amazon ECS when inference should match the surroundings you already run

If you would like inference to seem like each different service you use, operating throughout the similar orchestration, networking, and observability stack, this sample matches. When you already standardize on Kubernetes or Amazon ECS, package deal the Unsloth runtime into that surroundings as an alternative of introducing a separate serving floor just for fashions.

This sample is especially helpful when:

• Inference must reside subsequent to different software companies.
• You have already got robust tooling round Amazon EKS or Amazon ECS.
• You’ve got already standardized deployment, networking, observability, and safety on the infrastructure layer.

In these environments, the sensible query turns into much less “Ought to I take advantage of Amazon SageMaker AI?” and extra “Which Unsloth export most closely fits the runtime I need to containerize?”

For groups that want managed cluster infrastructure for large-scale mannequin coaching and inference, Amazon SageMaker HyperPod can also be price evaluating, although this put up focuses on inference-only deployment patterns.

Operational practices that matter most

Irrespective of which AWS path you select, a number of practices have an outsized impact on whether or not the deployment works properly in manufacturing. For detailed steerage on every subject, see the Amazon SageMaker AI internet hosting documentation.

Hold immediate formatting constant finish to finish

Inconsistent immediate formatting is likely one of the commonest methods to misdiagnose a great mannequin as a foul deployment. When an exported mannequin behaves worse exterior Unsloth, the issue is usually not the quantization methodology. It’s normally a mismatch within the chat template, end-of-sequence (EOS) dealing with (the logic that tells the mannequin when to cease producing tokens), or immediate construction between coaching and inference. If the mannequin is coherent in a single surroundings and unstable in one other, verify immediate formatting earlier than investigating the mannequin file. See the Unsloth chat templates documentation for particulars on export-time template configuration.

Benchmark the total deployment form, not solely the quantization stage

A quantization benchmark by itself isn’t sufficient. Actual conduct is dependent upon context size, concurrency, request combine, streaming conduct, startup path, and the runtime itself. A smaller mannequin file may not produce the perfect manufacturing consequence if the applying requires longer contexts or burstier visitors. Likewise, a extra correct mannequin file may not be price the additional reminiscence price if the workload is latency-sensitive and cost-constrained. Consider the total deployment image: file measurement, runtime conduct, {hardware} match, and request sample. Amazon SageMaker AI additionally gives inference suggestions that may make it easier to establish optimum occasion varieties and configurations.

Use steady artifact supply

After you finalize the mannequin, put it on the market by means of a predictable storage path. On AWS, that normally means storing the chosen mannequin file in Amazon S3 and loading it from there quite than relying on runtime downloads from exterior sources throughout startup. That makes endpoint initialization extra predictable and simplifies governance, particularly in restricted community environments.

Monitor the service, not solely the mannequin

Manufacturing efficiency will not be solely about token high quality. It’s additionally about startup time, tail latency, concurrency, scaling conduct, and model-loading traits. Managed endpoints make this extra seen as a result of you’ll be able to watch invocation metrics, concurrency conduct, and scaling alerts instantly by means of Amazon CloudWatch. On self-managed Amazon EC2 deployments, expose and observe the equal alerts.

Validate the container contract early

When you’re deploying on Amazon SageMaker AI with a customized container, validate the container regionally earlier than you deploy it. Make sure that /ping and /invocations behave accurately, the mannequin hundreds from the anticipated path, and the runtime survives real looking immediate sizes. That step catches many integration points earlier than they flip into endpoint debugging classes. See the Amazon SageMaker customized container information for the total interface specification.

Design for safety and networking from the beginning

Mannequin serving typically turns into half of a bigger software surroundings, which suggests networking, IAM position scoping, web entry, AWS Key Administration Service (AWS KMS) encryption, and VPC placement matter as a lot as runtime efficiency. Make these selections up entrance quite than retrofitting them after the endpoint is already in use. Allow AWS CloudTrail for API-level audit logging throughout deployed sources. See the Amazon SageMaker infrastructure safety documentation for VPC configuration and encryption steerage.

Conclusion

The easiest way to deploy Unsloth-quantized fashions on AWS is to start with the mannequin file you really need to serve. GGUF provides you a light-weight path into llama.cpp and llama-server, which makes Amazon EC2 essentially the most direct place to validate a mannequin and Amazon SageMaker AI a sensible subsequent step while you want managed endpoints. Merged weights provide you with a extra pure path into vLLM, SGLang, and LMI containers when throughput and scale matter greater than a minimal runtime.

Though this put up makes use of Qwen3-VL-8B-Instruct as the instance, Unsloth helps a variety of open-source mannequin architectures together with Llama, Qwen, Mistral, Gemma, and others. The deployment patterns apply no matter which mannequin you select.

That’s the actual benefit of mixing Unsloth with AWS. You may regulate for price, throughput, or surroundings match with out altering the core mannequin workflow. Begin with the mannequin file, align it to the runtime, after which select the AWS service that matches the way you need to function inference in manufacturing.

Subsequent steps

To see these patterns in motion, clone the companion GitHub repository and run terraform apply to deploy each a quantized GGUF endpoint and a full-precision vLLM endpoint on Amazon SageMaker AI. Terraform provisions the infrastructure and an Amazon SageMaker Pocket book Occasion with the repository pre-cloned, so you can begin operating the comparability pocket book instantly.

The pocket book sends similar picture prompts to each endpoints, demonstrating multimodal vision-language inference with Qwen3-VL, and produces side-by-side comparisons of output high quality, latency, throughput, and price. It additionally features a benchmarking framework that evaluates each fashions towards a structured dataset utilizing goal high quality metrics (precise match, BLEU, ROUGE-L). With this framework, you’ll be able to quantify degradation from quantization quite than counting on subjective judgment. For automated occasion choice and configuration tuning past what the pocket book measures, Amazon SageMaker AI additionally gives inference suggestions that may establish optimum deployment configurations in your workload. To keep away from ongoing prices if you find yourself carried out, run terraform destroy to take away the provisioned sources.

To study extra:

In regards to the authors