Accelerating LLM inference with post-training weight and activation utilizing AWQ and GPTQ on Amazon SageMaker AI

Basis fashions (FMs) and enormous language fashions (LLMs) have been quickly scaling, typically doubling in parameter rely inside months, resulting in vital enhancements in language understanding and generative capabilities. This speedy development comes with steep prices: inference now requires huge reminiscence capability, high-performance GPUs, and substantial power consumption. This development is obvious within the open supply house. In 2023, TII-UAE launched Falcon 180B, the most important open mannequin on the time. Meta surpassed that in 2024 with Llama 3.1, a 405B dense mannequin. As of mid-2025, the most important publicly accessible mannequin is DeepSeek (V3 – Instruct variant, R1 – Reasoning variant), a combination of specialists (MoE) structure with 671 billion complete parameters—of which 37 billion are lively per token. These fashions ship state-of-the-art efficiency throughout a variety of duties, together with multi-modal search, code era, summarization, thought era, logical reasoning, and even PhD-level drawback fixing. Regardless of their worth, deploying such fashions in real-world purposes stays largely impractical due to their dimension, value, and infrastructure necessities.

We regularly depend on the intelligence of huge fashions for mission-critical purposes corresponding to customer-facing assistants, medical analysis, or enterprise brokers, the place hallucinations can result in severe penalties. Nonetheless, deploying fashions with over 100 billion parameters at scale is technically difficult—these fashions require vital GPU sources and reminiscence bandwidth, making it troublesome to spin up or scale down situations shortly in response to fluctuating consumer demand. In consequence, scaling to 1000’s of customers shortly turns into cost-prohibitive, as a result of the high-performance infrastructure necessities make the return on funding (ROI) troublesome to justify. Submit-training quantization (PTQ) affords a sensible different; by changing 16- or 32-bit weights and activations into lower-precision 8- or 4-bit integers after coaching, PTQ can shrink mannequin dimension by 2–8 occasions, cut back reminiscence bandwidth necessities, and pace up matrix operations, all with out the necessity for retraining, making it appropriate for deploying massive fashions extra effectively. For instance, the bottom DeepSeek-V3 mannequin requires an ml.p5e.48xlarge occasion (with 1128 GB H100 GPU reminiscence) for inference, whereas its quantized variant (QuixiAI/DeepSeek-V3-0324-AWQ) can run on smaller situations corresponding to ml.p5.48xlarge (with 640 GB H100 GPU reminiscence) and even ml.p4de.24xlarge (with 640 GB A100 GPU reminiscence). This effectivity is achieved by making use of low-bit quantization to much less influential weight channels, whereas preserving or rescaling the channels which have the best impression on activation responses, and protecting activations in full precision—dramatically decreasing peak reminiscence utilization.

Quantized fashions are made attainable by contributions from the developer neighborhood—together with tasks like Unsloth AI and QuixiAI (previously: Cognitive Computations)—that make investments vital time and sources into optimizing LLMs for environment friendly inference. These quantized fashions might be seamlessly deployed on Amazon SageMaker AI utilizing a number of traces of code. Amazon SageMaker Inference gives a totally managed service for internet hosting machine studying, deep studying, and enormous language or imaginative and prescient fashions at scale in a cheap and production-ready method. On this put up, we discover why quantization issues—the way it permits lower-cost inference, helps deployment on resource-constrained {hardware}, and reduces each the monetary and environmental impression of recent LLMs, whereas preserving most of their unique efficiency. We additionally take a deep dive into the rules behind PTQ and reveal the way to quantize the mannequin of your alternative and deploy it on Amazon SageMaker.

The steps are:

1. Select mannequin
2. Select W_xA_y method (W_xA_y right here implies weights and activations, which will likely be mentioned in depth later on this put up)
3. Select algorithm (AWQ, GPTQ, SmoothQuant, and so forth)
4. Quantize
5. Deploy and inference

For instance this workflow and assist visualize the method, we’ve included the next stream diagram.

Conditions

To run the instance notebooks, you want an AWS account with an AWS Identification and Entry Administration (IAM) position with permissions to handle sources created. For extra info, see Create an AWS account.

If that is your first time working with Amazon SageMaker Studio, you first have to create a SageMaker area.

By default, the mannequin runs in a shared AWS managed digital non-public cloud (VPC) with web entry. To reinforce safety and management entry, it is best to explicitly configure a personal VPC with acceptable safety teams and IAM insurance policies primarily based in your necessities.

Amazon SageMaker AI gives enterprise-grade security measures to assist maintain your information and purposes safe and personal. We don’t share your information with mannequin suppliers, offering you full management over your information. This is applicable to all fashions—each proprietary and publicly accessible, together with DeepSeek-R1 on SageMaker. For extra info, see Configure safety in Amazon SageMaker AI.

As a greatest observe, it’s at all times advisable to deploy your LLM’s endpoints inside your VPC and behind a personal subnet with out web gateways and ideally with no egress. Ingress from the web also needs to be blocked to reduce safety dangers.

On this put up, we use LiteLLM Python SDK to standardize and summary entry to Amazon SageMaker real-time endpoints and LLMPerf instrument for analysis of efficiency of our quantized fashions. See Set up within the LLMPerf GitHub repo for setup directions.

Weights and activation strategies (WₓAᵧ)

As the size of LLMs continues to develop, deploying them effectively turns into much less about uncooked efficiency and extra about discovering the suitable stability between pace, value, and accuracy. In real-world eventualities, quantization begins with three core concerns:

• The dimensions of the mannequin it is advisable to host
• The associated fee or goal {hardware} accessible for inference
• The appropriate trade-off between accuracy and inference pace

Understanding how these elements form quantization selections is essential to creating LLMs viable in manufacturing environments. We’ll discover how post-training quantization strategies like AWQ and generative pre-trained transformers quantization (GPTQ) assist navigate these constraints and make state-of-the-art fashions deployable at scale.

Weights and activation: A deep dive

In neural networks, weights are the static, realized parameters saved within the mannequin—consider them because the mounted coefficients that form how inputs are mixed—whereas activations are the dynamic values produced at every layer while you run information by way of the community, representing the response of every neuron to its inputs. The previous determine illustrates weights and activations in a mannequin stream. We seize their respective precisions with the shorthand WₓAᵧ, the place Wₓ is the bit-width for weights (for instance, 4-bit or 8-bit) and Aᵧ is the bit-width for activations (for instance, 8-bit or 16-bit). For instance, W4A16 means weights are saved as 4-bit integers (typically with per-channel, symmetric or uneven scaling) whereas activations stay in 16-bit floating level. This notation tells you which of them components of the mannequin are compressed and by how a lot, serving to you stability reminiscence use, compute pace, and accuracy.

W4A16 (or W4A16_symmetric)

W4A16 refers to 4-bit precision for weights and 16-bit for activations, utilizing a symmetric quantization for weights. Symmetric quantization means the quantizer’s vary is centered round zero (absolutely the minimal and most of the load distribution are set to be equal in magnitude). Utilizing 4-bit integer weights yields an 8-times discount in weight reminiscence in comparison with FP32 (or 4 occasions in comparison with FP16), which could be very enticing for deployment. Nonetheless, with solely 16 quantization ranges (−8 to +7 for a 4-bit signed integer, in a symmetric scheme), the mannequin is susceptible to quantization error. If the load distribution isn’t completely zero-centered (for instance, if weights have a slight bias or a number of massive outliers), a symmetric quantizer may waste vary on one aspect and never have sufficient decision the place the majority of values lie. Research have discovered {that a} naive 4-bit symmetric quantization of LLM weights can incur a noticeable accuracy drop and is usually inferior to utilizing an uneven scheme at this low bit-width. The symmetric W4A16 method is principally a baseline; with out extra strategies (like AWQ’s scaling or GPTQ’s error compensation), 4-bit weight quantization wants cautious dealing with to keep away from severe degradation.

W4A16_asymmetric

Utilizing 4-bit weights with an uneven quantization improves upon the symmetric case by introducing a zero-point offset. Uneven quantization maps the minimal weight to the bottom representable integer and the utmost weight to the best integer, reasonably than forcing the vary to be symmetric round zero. This enables the small 4-bit scale to cowl the precise vary of weight values extra successfully. In observe, 4-bit weight quantization with uneven scaling considerably outperforms the symmetric method when it comes to mannequin accuracy. By higher using all 16 ranges of the quantizer (particularly when the load distribution has a non-zero imply or distinguished outliers on one aspect), the uneven W4A16 scheme can cut back the quantization error. Trendy PTQ strategies for 4-bit LLMs nearly at all times incorporate some type of uneven or per-channel scaling for that reason. For instance, one method is group-wise quantization the place every group of weights (for instance, every output channel) will get its personal min-max vary—successfully an uneven quantization per group—which has been recognized as a sweet-spot when mixed with 4-bit weights. W4A16 with uneven quantization is the popular technique for pushing weights to ultra-low precision, as a result of it yields higher perplexity and accuracy retention than a symmetric 4-bit mapping.

W8A8

This denotes totally quantizing each weights and activations to 8-bit integers. INT8 quantization is a well-understood, extensively adopted PTQ method that often incurs minimal accuracy loss in lots of networks, as a result of 256 distinct ranges (per quantization vary) are often adequate to seize the wanted precision. For LLMs, weight quantization to 8-bit is comparatively simple—analysis has proven that changing 16-bit weights with INT8 typically causes negligible change in perplexity. Activation quantization to 8-bit, nevertheless, is more difficult for transformers due to the presence of outliers—occasional very massive activation values in sure layers. These outliers can drive a quantizer to have a particularly massive vary, making most values use solely a tiny fraction of the 8-bit ranges (leading to precision loss). To handle this, strategies like SmoothQuant redistribute among the quantization problem from activations to weights—primarily cutting down outlier activation channels and scaling up the corresponding weight channels (a mathematically equal transformation) in order that activations have a tighter vary that matches nicely in 8 bits. With such calibrations, LLMs might be quantized to W8A8 with little or no efficiency drop. The advantage of W8A8 is that it permits end-to-end integer inference—each weights and activations are integers—which present {hardware} can exploit for quicker matrix multiplication. Absolutely INT8 fashions typically run quicker than combined precision fashions, as a result of they’ll use optimized INT8 arithmetic all through.

W8A16

W8A16 makes use of 8-bit quantization for weights whereas protecting activations in 16-bit precision (typically FP16). It may be seen as a weight-only quantization state of affairs. The reminiscence financial savings from compressing weights to INT8 are vital (a 2 occasions discount in comparison with FP16, and 4 occasions in comparison with FP32) and, as famous, INT8 weights often don’t harm accuracy in LLMs. As a result of activations stay in excessive precision, the mannequin’s computation outcomes are practically as correct as the unique—the principle supply of error is the minor quantization noise in weights. Weight-only INT8 quantization is thus a really protected alternative that yields substantial reminiscence discount with nearly no mannequin high quality loss.

Many sensible deployments begin with weight-only INT8 PTQ as a baseline. This method is very helpful while you wish to cut back mannequin dimension to suit on a tool inside a given reminiscence finances with out doing advanced calibration for activations. By way of pace, utilizing INT8 weights reduces reminiscence bandwidth necessities (benefiting memory-bound inference eventualities) and might barely enhance throughput, nevertheless the activations are nonetheless 16-bit, and the compute items won’t be totally using integer math for accumulation. If the {hardware} converts INT8 weights to 16-bit on the fly to multiply by FP16 activations, the pace achieve is perhaps restricted by that conversion. For memory-bound workloads (widespread with LLMs at small batch sizes), INT8 weights present a noticeable speed-up as a result of the bottleneck is commonly fetching weights from reminiscence. For compute-bound eventualities (corresponding to very massive batch throughput), weight-only quantization alone yields much less profit—in these circumstances, you may quantize activations (transferring to W8A8) to make use of quick INT8×INT8 matrix multiplication totally. In abstract, W8A16 is simple to implement quantization scheme that dramatically cuts mannequin dimension with minimal danger, whereas W8A8 is the following step to maximise inference pace at the price of a extra concerned calibration course of.

Abstract

The next desk gives a high-level overview of the WₓAᵧ paradigm.

Inference acceleration by way of PTQ strategies

As outlined earlier, LLMs with excessive parameter counts are extraordinarily resource-intensive at inference. Within the following sections, we discover how PTQ reduces these necessities, enabling cheaper and performant inference. As an example, a Llama 3 70B parameter mannequin at FP16 precision doesn’t match right into a single A100 80 GB GPU and requires at the least two A100 80 GB GPUs for affordable inference at scale, making deployment each expensive and impractical for a lot of use circumstances. To handle this problem, PTQ converts a educated mannequin’s weights (and typically activations) from high-precision floats (for instance, 16- or 32-bit) to lower-bit integers (for instance, 8-bit or 4-bit) after coaching. This compression can shrink mannequin dimension by 2–8 occasions, enabling the mannequin to slot in reminiscence and decreasing reminiscence bandwidth calls for, which in flip can pace up inference.

Crucially, PTQ requires no extra coaching—in contrast to quantization-aware coaching (QAT), which contains quantization into the fine-tuning course of. PTQ avoids the prohibitive retraining value related to billion-parameter fashions. The problem is to quantize the mannequin rigorously to reduce any drop in accuracy or improve in perplexity. Trendy PTQ strategies try to retain mannequin efficiency whereas dramatically bettering deployment effectivity.

Submit-training quantization algorithms

Quantizing a complete mannequin on to 4-bit or 8-bit precision might sound simple, however doing so naïvely typically leads to substantial accuracy degradation—significantly underneath lower-bit configurations. To beat this, specialised PTQ algorithms have been developed that intelligently compress mannequin parameters whereas preserving constancy. On this put up, we give attention to two extensively adopted and well-researched PTQ strategies, every taking a definite method to high-accuracy compression:

• Activation-aware weights quantization (AWQ)
• Generative pre-trained transformers quantization (GPTQ)

Activation conscious weights quantization

AWQ is a PTQ method that targets weight-only quantization at very low bit widths (usually 4-bit) whereas protecting activations in greater precision, corresponding to FP16. The core idea is that not all weights contribute equally to a mannequin’s output; a small subset of salient weights disproportionately influences predictions. By figuring out and preserving roughly 1% of those important weight channels—these related to the most important activation values—AWQ can dramatically shut the hole between 4-bit quantized fashions and their unique FP16 counterparts when it comes to perplexity. In contrast to conventional strategies that rank significance primarily based on weight magnitude alone, AWQ makes use of activation distributions to seek out which weights actually matter. Early outcomes confirmed that leaving the highest 1% of channels in greater precision was sufficient to keep up efficiency—however this introduces {hardware} inefficiencies because of mixed-precision execution. To get round this, AWQ introduces a chic workaround of per-channel scaling.

Throughout quantization, AWQ amplifies the weights of activation-salient channels to cut back relative quantization error and folds the inverse scaling into the mannequin, so no specific rescaling is required throughout inference. This adjustment eliminates the overhead of mixed-precision computation whereas protecting inference purely low-bit. Importantly, AWQ achieves this with out retraining—it makes use of a small calibration dataset to estimate activation statistics and derive scaling elements analytically. The strategy avoids overfitting to calibration information, guaranteeing sturdy generalization throughout duties. In observe, AWQ delivers near-FP16 efficiency even at 4-bit precision, displaying far smaller degradation than conventional post-training strategies like RTN (round-to-nearest). Whereas there’s nonetheless a marginal improve in perplexity in comparison with full-precision fashions, the trade-off is commonly negligible given the three–4 occasions discount in reminiscence footprint and bandwidth. This effectivity permits deployment of very massive fashions—as much as 70 billion parameters—on a single high-end GPU corresponding to an A100 or H100. In brief, AWQ demonstrates that with cautious, activation-aware scaling, precision might be centered the place it issues most, attaining low-bit quantization with minimal impression on mannequin high quality.

Generative pre-trained transformers quantization (GPTQ)

GPTQ is one other PTQ methodology that takes an error-compensation-driven method to compressing massive language fashions. GPTQ operates layer by layer, aiming to protect every layer’s output as carefully as attainable to that of the unique full-precision mannequin. It follows a grasping, sequential quantization technique: at every step, a single weight or a small group of weights is quantized, whereas the remaining unquantized weights are adjusted to compensate for the error launched. This retains the output of every layer tightly aligned with the unique. The method is knowledgeable by approximate second-order statistics, particularly an approximation of the Hessian matrix, which estimates how delicate the output is to modifications in every weight. This optimization process is typically known as optimum mind quantization, the place GPTQ rigorously quantizes weights in an order that minimizes cumulative output error.

Regardless of its sophistication, GPTQ stays a one-shot PTQ methodology—it doesn’t require retraining or iterative fine-tuning. It makes use of a small calibration dataset to run ahead passes, gathering activation statistics and estimating Hessians, however avoids any weight updates past the grasping compensation logic. The result’s an impressively environment friendly compression method: GPTQ can quantize fashions to three–4 bits per weight with minimal accuracy loss, even for large fashions. For instance, the strategy demonstrated compressing a 175 billion-parameter GPT mannequin to three–4 bits in underneath 4 GPU-hours, with negligible improve in perplexity, enabling single-GPU inference for the primary time at this scale. Whereas GPTQ delivers excessive accuracy, its reliance on calibration information has led some researchers to notice gentle overfitting results, particularly for out-of-distribution inputs. Nonetheless, GPTQ has develop into a go-to baseline in LLM quantization due to its sturdy stability of constancy and effectivity, aided by mathematical optimizations corresponding to quick Cholesky-based Hessian updates that make it sensible even for fashions with tens or a whole lot of billions of parameters.

Utilizing Amazon SageMaker AI for inference optimization and mannequin quantization

On this part, we cowl the way to implement quantization utilizing Amazon SageMaker AI. We stroll by way of a codebase that you should use to shortly quantize a mannequin utilizing both the GPTQ or AWQ methodology on SageMaker coaching jobs backed by a number of GPU situations. The code makes use of the open supply vllm-project/llm-compressor package deal to quantize dense LLM weights from FP32 to INT4.

All code for this course of is out there within the amazon-sagemaker-generativeai GitHub repository. The llm-compressor challenge gives a streamlined library for mannequin optimization. It helps a number of algorithms—GPTQ, AWQ, and SmoothQuant—for changing full- or half-precision fashions into lower-precision codecs. Quantization takes place in three steps, described within the following sections. The total implementation is out there in post_training_sagemaker_quantizer.py, with arguments offered for simple execution.

Step 1: Load mannequin utilizing HuggingFace transformers

Load the mannequin weights with out attaching them to an accelerator. The llm-compressor library routinely detects accessible {hardware} and offloads weights to the accelerator as wanted. As a result of it performs quantization layer by layer, your complete mannequin doesn’t want to slot in accelerator reminiscence without delay.

def quantize_model(
    args: argparse.Namespace
) -> None:
    attempt:

        ...
        # load mannequin
        mannequin = AutoModelForCausalLM.from_pretrained(
            args.model_id,
            torch_dtype="auto",
            device_map=None,
            trust_remote_code=True
        )
        # load tokenizer
        tokenizer_or_processor = AutoTokenizer.from_pretrained(
            args.model_id,
            trust_remote_code=True
        )
       ...

Step 2: Choose and cargo the calibration dataset

A calibration dataset is used throughout PTQ to estimate activation ranges and statistical distributions in a pretrained LLM with out retraining. Instruments like llm-compressor use this small, consultant dataset to run ahead passes and accumulate statistics corresponding to minimal and most values or percentiles. These statistics information the quantization of weights and activations to cut back precision whereas preserving mannequin accuracy. You should use any tokenized dataset that displays the mannequin’s anticipated enter distribution for calibration.

from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQModifier
from llmcompressor.modifiers.quantization import GPTQModifier
....

def preprocess_data(
    dataset: Any,
    tokenizer: AutoTokenizer,
    max_sequence_length: int
) -> Any:
    def preprocess(instance):
        return {
            "textual content": tokenizer.apply_chat_template(
                instance["messages"],
                tokenize=False,
            )
        }

    def tokenize(pattern: Dict) -> Dict:
        return tokenizer(
            pattern["text"],
            padding=False,
            max_length=max_sequence_length,
            truncation=True,
            add_special_tokens=False,
        )

    dataset = dataset.map(preprocess)
    dataset = dataset.map(tokenize,  remove_columns=dataset.column_names)
    return dataset

Step 3: Run PTQ on the candidate mannequin

The oneshot methodology in llm-compressor performs a single-pass (no iterative retraining) PTQ utilizing a specified recipe, making use of each weight and activation quantization (and optionally sparsity) in a single cross.

• num_calibration_samples defines what number of enter sequences (for instance, 512) are used to simulate mannequin habits, gathering the activation statistics obligatory for calibrating quantization ranges.
• max_seq_length units the utmost token size (for instance, 2048) for these calibration samples, so activations mirror the worst-case sequence context, guaranteeing quantization stays correct throughout enter lengths.

Collectively, these hyperparameters management the representativeness and protection of calibration, instantly impacting quantization constancy.

The modifier courses (GPTQModifier, AWQModifier) settle for a schema parameter that defines the bit-width for each weights and activations. By way of this parameter, you may specify codecs corresponding to W8A8 (8-bit weights and activations) or W4A16 (4-bit weights with 16-bit activations), providing you with fine-grained management over precision trade-offs throughout mannequin layers.

        ...
        ... 
        logger.information(f"Configuring {args.algorithm.higher()} quantization")
        if args.algorithm == "awq":

            quant_scheme = args.awq_quantization_scheme
            recipe = [
                AWQModifier(
                    ignore=[val.rstrip() for val in args.ignore_layers.split(',')],
                    scheme=args.awq_quantization_scheme,
                    targets=[val.rstrip() for val in args.include_targets.split(',')]
                )
            ]

        ...
        elif args.algorithm == "gptq":

            quant_scheme = args.gptq_quantization_scheme
            recipe = [
                GPTQModifier(
                    ignore=[val.rstrip() for val in args.ignore_layers.split(',')],
                    scheme=args.gptq_quantization_scheme,
                    targets=[val.rstrip() for val in args.include_targets.split(',')]
                )
            ]
       ...
       ...
        oneshot(
            mannequin=mannequin,
            dataset=processed_dataset,
            recipe=recipe,
            max_seq_length=args.max_sequence_length, # <- Set max sequence size
            num_calibration_samples=args.num_calibration_samples, # <- Set max calibration - variety of iterations of stats calculation
            output_dir=save_dir,
            trust_remote_code_model=True
        )

Structure sample for quantization on Amazon SageMaker AI

Your complete workflow, proven within the following determine, is applied within the post_training_sagemaker_quantizer.py script and might be executed as a SageMaker coaching job on an occasion with NVIDIA GPU assist (corresponding to ml.g5.2xlarge) for accelerated quantization.

This course of doesn’t contain coaching or fine-tuning the mannequin. The coaching job is used solely to run PTQ with GPU acceleration.

...
hyperparameters = {
    'model-id': 'meta-llama/Llama-3.1-8B-Instruct',
    'dataset-id': 'HuggingFaceH4/ultrachat_200k',
    'dataset-split': 'train_sft',
    'dataset-seed': 42,
    'algorithm': 'gptq',
    'max-sequence-length': 1024,
    'num-calibration-samples': 256,
    'ignore-layers': 'lm_head',
    'include-targets': 'Linear',
    'gptq-quantization-scheme': 'W8A8',
}

quantization_estimator = PyTorch(
    entry_point="post_training_sagemaker_quantizer.py",
    source_dir="./scripts",
    instance_type="ml.g6e.2xlarge",
    instance_count=1,
    position=position,
    framework_version='2.4.0',
    py_version='py311',
    hyperparameters=hyperparameters,
    surroundings={"HF_TOKEN": "my-awesome-hf-token"}
)
...

After a mannequin is quantized, will probably be saved to Amazon Easy Storage Service (Amazon S3) instantly as an output from the SageMaker coaching job. We’ll uncompress the mannequin and host it as a SageMaker real-time endpoint utilizing a Amazon SageMaker AI massive mannequin inference (LMI) container, powered by vLLM. To seek out the most recent photographs, see AWS Deep Studying Framework Assist Coverage for LMI containers (see SageMaker part).

...

prebaked_inference_image_uri = f"763104351884.dkr.ecr.{sagemaker.Session().boto_session.region_name}.amazonaws.com/djl-inference:0.33.0-lmi15.0.0-cu128"
...
quant_model = sagemaker.Mannequin(
    image_uri=prebaked_inference_image_uri,
    env={
        "HF_MODEL_ID": f"{remote_upload_s3uri}/", <- Your mannequin S3 path
        "OPTION_MAX_MODEL_LEN": "12000",
        "OPTION_GPU_MEMORY_UTILIZATION": "0.95",
        "OPTION_ENABLE_STREAMING": "false",
        "OPTION_ROLLING_BATCH": "auto",
        "OPTION_MODEL_LOADING_TIMEOUT": "3600",
        "OPTION_PAGED_ATTENTION": "false",
        "OPTION_DTYPE": "fp16",
    },
    position=position,
    title=model_name,
    sagemaker_session=sagemaker.Session()
)
...
pretrained_predictor = quant_model.deploy(
    endpoint_name=endpoint_name,
    initial_instance_count=1,
    instance_type="ml.g5.2xlarge",
    container_startup_health_check_timeout=600,
    wait=False
)
print(f"Your Endpoint: {endpoint_name} is now deployed!")
```

You now have a SageMaker real-time endpoint serving your quantized mannequin and prepared for inference. You’ll be able to question it utilizing the SageMaker Python SDK or litellm, relying in your integration wants.

 from litellm import completion

response = completion(
        mannequin=f"sagemaker/{endpoint_name}", 
        messages=[{ "content": "Hello", "role": "user"}, { "content": "You are a helpful assistant that follows instructions", "role": "system"}],
        temperature=0.1,
        max_tokens=64
    )

Mannequin efficiency

We are going to use an ml.g5.2xlarge occasion for Llama-3.1-8B and Qwen-2.5-VL-7B fashions and ml.p4d.24xlarge occasion for Llama-3.1-70B mannequin and an LMI container v15 with vLLM backend as a serving framework.

The next is a code snippet from the deployment configuration:

lmi_env = {
    "SERVING_FAIL_FAST": "true",
    "OPTION_ASYNC_MODE": "true",
    "OPTION_ROLLING_BATCH": "disable",
    "OPTION_MAX_MODEL_LEN": "8192",
    "OPTION_TENSOR_PARALLEL_DEGREE": "max",
    "OPTION_ENTRYPOINT": "djl_python.lmi_vllm.vllm_async_service",
}

This efficiency analysis’s major objective is to indicate the relative efficiency of mannequin variations on totally different {hardware}. The combos aren’t totally optimized and shouldn’t be considered as peak mannequin efficiency on an occasion kind. At all times ensure that to check utilizing your information, visitors, and I/O sequence size. The next is efficiency benchmark script:

#!/bin/bash
export LLM_PERF_CONCURRENT=1
export LLM_PERF_MAX_REQUESTS=$(expr $LLM_PERF_CONCURRENT * 10)
export LLM_PERF_SCRIPT_DIR=$HOME/5_projects/llmperf

export LLM_PERF_OUTPUT=outputs/test-2025-07-08-21-45-57-221

mkdir -p $LLM_PERF_OUTPUT
cp "$0" "${LLM_PERF_OUTPUT}"/

python3 ${LLM_PERF_SCRIPT_DIR}/token_benchmark_ray.py 
    --model "sagemaker/model-2025-07-08-21-01-10-147" 
    --mean-input-tokens 512 
    --stddev-input-tokens 32 
    --mean-output-tokens 256 
    --stddev-output-tokens 16 
    --max-num-completed-requests ${LLM_PERF_MAX_REQUESTS} 
    --timeout 1800 
    --num-concurrent-requests ${LLM_PERF_CONCURRENT} 
    --results-dir "${LLM_PERF_OUTPUT}" 
    --llm-api litellm 
    --additional-sampling-params '{}'

Efficiency metrics

To grasp the impression of PTQ optimization strategies, we give attention to 5 key inference efficiency metrics—every providing a distinct lens on system effectivity and consumer expertise:

• GPU reminiscence utilization: Signifies the proportion of complete GPU reminiscence actively used throughout inference. Larger reminiscence utilization suggests extra of the mannequin or enter information is loaded into GPU reminiscence, which might enhance throughput—however extreme utilization may result in reminiscence bottlenecks or out-of-memory errors.
• Finish-to-end latency: Measures the entire time taken from enter submission to ultimate output. That is important for purposes the place responsiveness is essential, corresponding to real-time methods or user-facing interfaces.
• Time to first token (TTFT): Captures the delay between enter submission and the era of the primary token. Decrease TTFT is very necessary for streaming or interactive workloads, the place perceived responsiveness issues greater than complete latency.
• Inter-token latency (ITL): Tracks the typical time between successive token outputs. A decrease ITL leads to smoother, faster-seeming responses, significantly in long-form textual content era.
• Throughput: Measures the variety of tokens generated per second throughout all concurrent requests. Larger throughput signifies higher system effectivity and scalability, enabling quicker processing of huge workloads or extra simultaneous consumer periods.

Collectively, these metrics present a holistic view of inference habits—balancing uncooked effectivity with real-world usability. Within the subsequent sections of this put up, we consider three candidate fashions—every various in dimension and structure—to validate inference efficiency metrics after quantization utilizing AWQ and GPTQ algorithms throughout totally different WₓAᵧ methods. The chosen fashions embrace:

• Llama-3.1-8B-Instruct: An 8-billion parameter dense decoder-only transformer mannequin optimized for instruction following. Printed by Meta, it belongs to the LLaMA (Giant Language Mannequin Meta AI) household and is well-suited for general-purpose pure language processing (NLP) duties.
• Llama-3.3-70B-Instruct: A 70-billion parameter mannequin additionally from Meta’s LLaMA collection, this bigger variant affords considerably improved reasoning and factual grounding capabilities, making it very best for high-performance enterprise use circumstances.
• Qwen2.5-VL-7B-Instruct: A 7-billion parameter vision-language mannequin developed by Alibaba’s Institute for Clever Computing. It helps each textual content and picture inputs, combining a transformer-based textual content spine with a visible encoder, making it appropriate for multimodal purposes.

Notice that every mannequin was examined on a distinct occasion kind: Llama-3.1-8B on ml.g5.2xlarge, Llama-3.3-70B on ml.p4dn.24xlarge, and Qwen2.5-VL-7B on ml.g6e.4xlarge.

GPU reminiscence utilization

GPU reminiscence utilization displays how a lot system reminiscence is consumed throughout mannequin execution and instantly impacts deployability, batch dimension, and {hardware} choice. Decrease reminiscence utilization permits operating bigger fashions on smaller GPUs or serving extra concurrent requests on the identical {hardware}. Quantization improves compute effectivity and considerably reduces the reminiscence footprint of LLMs. By changing high-precision weights (for instance, FP16 or FP32) into lower-bit codecs corresponding to INT8 or FP8, each AWQ and GPTQ methods allow fashions to devour considerably much less GPU reminiscence throughout inference. That is important for deploying massive fashions on memory-constrained {hardware} or growing batch sizes for greater throughput. Within the following desk and chart, we record and visualize the GPU reminiscence utilization (in GB) throughout the fashions underneath a number of quantization configurations. The share discount is in contrast in opposition to the bottom (unquantized) mannequin dimension, highlighting the reminiscence financial savings achieved with every WₓAᵧ technique, which ranges from ~30%–70% much less GPU reminiscence utilization after PTQ.

The determine beneath illustrates the GPU reminiscence footprint (in GB) of the mannequin in its uncooked (unquantized) kind in comparison with its quantized variants. Quantization leads to ~30%–70% discount in GPU reminiscence consumption, considerably decreasing the general reminiscence footprint.

Finish-to-end latency

Finish-to-end latency measures the entire time taken from the second a immediate is acquired to the supply of the ultimate output token. It’s a important metric for evaluating user-perceived responsiveness and general system efficiency, particularly in real-time or interactive purposes.

Within the following desk, we report end-to-end latency in seconds throughout various concurrency ranges (C=1 to C=128) for three fashions of various dimension and modality (Llama-3.1-8B, Llama-3.3-70B, and Qwen2.5-VL-7B) underneath totally different quantization methods.

The next graphs displaying finish to finish latency for various concurrency ranges for various fashions.

The determine above presents the end-to-end latency of the Llama 3-8B mannequin in its uncooked (unquantized) kind and its quantized variants throughout concurrency ranges starting from 1 to 128 on the identical occasion.

The determine above presents the end-to-end latency of the Qwen 2.7-7B mannequin in its uncooked (unquantized) kind and its quantized variants throughout concurrency ranges starting from 1 to 128 on the identical occasion.

The determine above presents the end-to-end latency of the Llama 3-70B mannequin in its uncooked (unquantized) kind and its quantized variants throughout concurrency ranges starting from 1 to 128 on the identical occasion.

Time to first token

TTFT measures the delay between immediate submission and the era of the primary token. This metric performs a vital position in shaping perceived responsiveness—particularly in chat-based, streaming, or interactive purposes the place preliminary suggestions time is important. Within the following desk, we examine TTFT in seconds for 3 fashions of various dimension and modality—Llama-3.1-8B, Llama-3.3-70B, and Qwen2.5-VL-7B—underneath totally different quantization methods. As concurrency will increase (from C=1 to C=128), the outcomes spotlight how quantization strategies like AWQ and GPTQ assist preserve low startup latency, guaranteeing a smoother and quicker expertise even underneath excessive load.

Inter-token latency

ITL measures the typical time delay between the era of successive tokens. It instantly impacts the smoothness and pace of streamed outputs—significantly necessary in purposes involving long-form textual content era or voice synthesis, the place delays between phrases or sentences can degrade consumer expertise. Within the following desk, we analyze ITL in seconds throughout three fashions of various dimension and modality—Llama-3.1-8B, Llama-3.3-70B, and Qwen2.5-VL-7B—underneath totally different quantization schemes. As concurrency scales up, the outcomes illustrate how quantization methods like AWQ and GPTQ assist preserve low per-token latency, guaranteeing fluid era even underneath excessive parallel masses.

Throughput

Throughput measures the variety of tokens generated per second and is a key indicator of how effectively a mannequin can scale underneath load. Larger throughput instantly permits quicker batch processing and helps extra concurrent consumer periods. Within the following desk, we current throughput outcomes for Llama-3.1-8B, Llama-3.3-70B, and Qwen2.5-VL-7B throughout various concurrency ranges and quantization methods. Quantized fashions preserve—and in lots of circumstances enhance—throughput, due to decreased reminiscence bandwidth and compute necessities. The substantial reminiscence financial savings from quantization permits a number of mannequin employees to be deployed on a single GPU, significantly on high-memory situations. This multi-worker setup additional amplifies complete system throughput at greater concurrency ranges, making quantization a extremely efficient technique for maximizing utilization in manufacturing environments.

Conclusion

Submit-training quantization (PTQ) strategies like AWQ and GPTQ have confirmed to be efficient options for deploying basis fashions in manufacturing environments. Our complete testing throughout totally different mannequin sizes and architectures demonstrates that PTQ considerably reduces GPU reminiscence utilization. The advantages are evident throughout all key metrics, with quantized fashions displaying higher throughput and decreased latency in inference time, together with high-concurrency eventualities. These enhancements translate to decreased infrastructure prices, improved consumer expertise by way of quicker response occasions, and the pliability of deploying bigger fashions on resource-constrained {hardware}. As language fashions proceed to develop in scale and complexity, PTQ affords a dependable method for balancing efficiency necessities with infrastructure constraints, offering a transparent path to environment friendly, cost-effective AI deployment.

On this put up, we demonstrated the way to streamline LLM quantization utilizing Amazon SageMaker AI and the llm-compressor module. The method of changing a full-precision mannequin to its quantized variant requires only a few easy steps, making it accessible and scalable for manufacturing deployments. By utilizing the managed infrastructure of Amazon SageMaker AI, organizations can seamlessly implement and serve quantized fashions for real-time inference, simplifying the journey from growth to manufacturing. To discover these quantization strategies additional, check with our GitHub repository.

Particular due to everybody who contributed to this text: Giuseppe Zappia, Dan Ferguson, Frank McQuillan and Kareem Syed-Mohammed.

Concerning the authors

Pranav Murthy is a Senior Generative AI Knowledge Scientist at AWS, specializing in serving to organizations innovate with Generative AI, Deep Studying, and Machine Studying on Amazon SageMaker AI. Over the previous 10+ years, he has developed and scaled superior pc imaginative and prescient (CV) and pure language processing (NLP) fashions to sort out high-impact issues—from optimizing international provide chains to enabling real-time video analytics and multilingual search. When he’s not constructing AI options, Pranav enjoys enjoying strategic video games like chess, touring to find new cultures, and mentoring aspiring AI practitioners. You will discover Pranav on LinkedIn

Dmitry Soldatkin is a Senior AI/ML Options Architect at Amazon Net Companies (AWS), serving to prospects design and construct AI/ML options. Dmitry’s work covers a variety of ML use circumstances, with a major curiosity in Generative AI, deep studying, and scaling ML throughout the enterprise. He has helped firms in lots of industries, together with insurance coverage, monetary providers, utilities, and telecommunications. You’ll be able to join with Dmitry on LinkedIn.