Orchestrating SageMaker AI endpoint auto scaling with AWS CDK

Updating the Amazon Sagemaker AI real-time endpoint configuration (altering the occasion sort, occasion rely, or surroundings variables) causes deployments by means of AWS CloudFormation or the AWS Cloud Growth Package (AWS CDK) to fail. SageMaker AI requires you to deregister the scalable goal earlier than updating the endpoint, however if you deregister it, CloudFormation can’t monitor the AWS Utility Auto Scaling assets.

On this submit, we present you the right way to use a reusable CDK assemble with AWS CloudFormation customized assets to orchestrate deregister and register auto scaling robotically throughout endpoint updates. You possibly can replace your endpoint configuration and auto scaling parameters in a single deployment with out handbook intervention.

Why this downside happens

Whenever you deploy a SageMaker AI endpoint with native CloudFormation auto scaling assets, then change the endpoint configuration and redeploy, a number of failure modes emerge. The particular failure is dependent upon what you modify and the way you’re employed across the problem.

Altering the endpoint configuration with auto scaling registered

CloudFormation tries to replace the endpoint whereas the scalable goal continues to be registered. SageMaker AI rejects the replace with the next error:

The variant(s) "[AllTraffic]" should be deregistered as scalable targets with Utility Auto Scaling earlier than they are often eliminated or have their occasion sort up to date.

SageMaker AI blocks the endpoint replace earlier than CloudFormation reaches the auto scaling assets.

Manually deregistering auto scaling, then altering solely the endpoint configuration

When you deregister the scalable goal by means of the AWS Command Line Interface (AWS CLI) and redeploy with solely an endpoint configuration change, the deployment succeeds. Nevertheless, CloudFormation doesn’t register auto scaling despite the fact that it’s within the CloudFormation template. As a result of the auto scaling parameters within the CloudFormation template haven’t modified, CloudFormation doesn’t register the scalable goal or re-apply the scaling coverage. The endpoint is now working with out auto scaling, and there’s no error to point something went fallacious.

Manually deregistering auto scaling, then altering auto scaling parameters

When you deregister the scalable goal by means of the AWS CLI and redeploy with auto scaling parameter adjustments, CloudFormation tries to replace the ScalableTarget useful resource that now not exists:

Useful resource of sort 'AWS::ApplicationAutoScaling::ScalableTarget' with identifier 'endpoint//variant/AllTraffic|sagemaker:variant:DesiredInstanceCount|sagemaker' was not discovered.

This failure is notable as a result of the rollback additionally fails (UPDATE_ROLLBACK_FAILED), leaving the stack in a damaged state that requires handbook intervention within the CloudFormation console to get well.

Resolution overview

You need to use this CDK assemble to interchange the native AWS::ApplicationAutoScaling::ScalableTarget and AWS::ApplicationAutoScaling::ScalingPolicy CloudFormation assets with three customized assets. These customized assets deregister and register auto scaling across the endpoint replace with specific dependency ordering, enforced by node.add_dependency calls in CDK. The total sequence is:

1. Deregister (customized useful resource): Removes the scalable goal earlier than the endpoint replace begins. On first deploy, no scalable goal exists but, so the handler handles the error gracefully and continues.
2. Endpoint replace (CDK-managed): CDK updates the endpoint useful resource usually, swapping to the brand new endpoint configuration.
3. Register Scalable Goal (customized useful resource): Waits for the endpoint to achieve InService standing, then registers the scalable goal.
4. Connect Scaling Coverage (customized useful resource): Waits for the scalable goal to be queryable, then applies the goal monitoring coverage.

Determine 1: Customized useful resource orchestration of SageMaker AI endpoint auto scaling round a CDK-managed endpoint replace.

Register Scalable Goal and Connect Scaling Coverage use the CDK supplier’s asynchronous completion sample. Every customized useful resource has an on_event handler that returns instantly and an is_complete handler. The supplier’s waiter re-invokes the is_complete handler on an interval (each 30 seconds) till its readiness verify passes. The supplier’s total_timeout (1 hour for Register Scalable Goal, quarter-hour for Connect Scaling Coverage) bounds the wait, so it isn’t restricted by the timeout of any single Lambda invocation.

A ConfigHash property on every customized useful resource forces CloudFormation to re-invoke the Lambda capabilities at any time when a configuration worth adjustments. The endpoint configuration and auto scaling parameters are outlined collectively as a result of a single ConfigHash drives the auto scaling orchestration.

Whereas the assemble defines three customized assets, the deployed stack comprises greater than three Lambda capabilities. Every customized useful resource is backed by a CDK cr.Supplier, and the supplier framework generates its personal helper Lambda capabilities. A synchronous supplier (Deregister) provides one framework operate. An asynchronous supplier (Register Scalable Goal and Connect Scaling Coverage) provides three (onEvent, isComplete, and onTimeout) plus an AWS Step Capabilities state machine that drives the waiter. Because of this, the deployed stack comprises three customized assets, two Step Capabilities state machines, and twelve Lambda capabilities. 5 of the assets maintain the assemble’s personal handler code (one for Deregister, and an on_event and is_complete pair for every asynchronous useful resource), with the remaining seven generated by the supplier framework. You don’t write or preserve the framework-generated capabilities, however they’re current within the stack and value accounting for if you evaluation the deployed assets.

Walkthrough

On this walkthrough, you deploy a SageMaker AI endpoint with auto scaling utilizing the offered instance stack, then confirm that configuration updates work accurately.

The walkthrough consists of the next steps:

1. Clone the repository and set up dependencies.
2. Configure your endpoint and auto scaling settings.
3. Deploy and confirm.
4. Check an replace.

The supply code is on the market within the linked GitHub repository.

Conditions

For this walkthrough, you want the next:

• An AWS account with permissions for Amazon SageMaker AI, AWS Lambda, AWS Id and Entry Administration (IAM), Utility Auto Scaling, and Amazon CloudWatch.
• AWS CDK v2 put in.
• Python 3.12 or later.
• uv for Python dependency administration.
• A SageMaker AI mannequin artifact in Amazon Easy Storage Service (Amazon S3) and a container picture in Amazon Elastic Container Registry (Amazon ECR), or a SageMaker AI built-in algorithm container.

Clone the repository and set up dependencies

1. Clone the repository:

   git clone https://github.com/aws-samples/sample-sagemaker-endpoint-autoscaling.git
   cd sample-sagemaker-endpoint-autoscaling

2. Set up dependencies:

Configure your endpoint and auto scaling settings

The endpoint and auto scaling settings are outlined collectively in config/config.yaml as a result of a single config_hash drives the auto scaling orchestration. A change to both part triggers the total sequence.

Edit config/config.yaml along with your values:

endpoint:
  endpoint_name: "my-model-endpoint"
  picture: "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-model:newest"
  model_data_url: "s3://my-bucket/fashions/mannequin.tar.gz"
  instance_type: "ml.m5.giant"
  initial_instance_count: 1
  surroundings: {}
autoscaling:
  min_capacity: 1
  max_capacity: 10
  target_value: 100
  scale_in_cooldown: 600
  scale_out_cooldown: 120

The endpoint parameters map to the SageMaker AI CreateModel and CreateEndpointConfig APIs. The auto scaling parameters map to the Utility Auto Scaling RegisterScalableTarget and PutScalingPolicy APIs.

Deploy and confirm

1. Bootstrap your AWS surroundings should you haven’t already:
2. Deploy the stack:

   uv run cdk deploy InferenceStack

CloudFormation creates the SageMaker AI mannequin, endpoint configuration, and endpoint, then runs the three customized assets in sequence to register auto scaling.

3. Confirm that the deployment registered the scalable goal:

   aws application-autoscaling describe-scalable-targets
   --service-namespace sagemaker
   --resource-ids endpoint/my-model-endpoint/variant/AllTraffic

4. Confirm that the deployment utilized the scaling coverage:

   aws application-autoscaling describe-scaling-policies
   --service-namespace sagemaker
   --resource-id endpoint/my-model-endpoint/variant/AllTraffic

Check an replace

Change a worth in config/config.yaml, for instance, replace the occasion sort from ml.m5.giant to ml.m5.xlarge, and redeploy:

uv run cdk deploy InferenceStack

After the deployment completes, confirm that auto scaling continues to be registered and has the up to date values:

aws application-autoscaling describe-scalable-targets
--service-namespace sagemaker
--resource-ids endpoint/my-model-endpoint/variant/AllTraffic

Integrating with an present stack

Use the instance stack should you’re beginning contemporary or desire a reference implementation. Use the assemble immediately if you have already got a CDK stack with SageMaker AI assets.

To combine with an present stack, copy the sagemaker_autoscaling/constructs/ listing into your mission and add the assemble alongside your endpoint assets:

import json
from sagemaker_autoscaling.constructs.sagemaker_autoscaling_construct import (
    AutoscalingConfig,
    SagemakerAutoscalingConstruct,
    SagemakerAutoscalingProps,
)

# Each endpoint and auto scaling values should be in config_hash in order that
# a change to both triggers the auto scaling orchestration.
config_hash = json.dumps({
    "endpoint": {
        "picture": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-model:newest",
        "modelDataUrl": "s3://my-bucket/fashions/mannequin.tar.gz",
        "instanceType": "ml.m5.giant",
        "initialInstanceCount": 1,
    },
    "autoscaling": {
        "minCapacity": 1,
        "maxCapacity": 10,
        "targetValue": 100,
        "scaleInCooldown": 600,
        "scaleOutCooldown": 120,
    },
})

SagemakerAutoscalingConstruct(
    self, "Autoscaling",
    props=SagemakerAutoscalingProps(
        endpoint=your_endpoint,  # your CfnEndpoint
        autoscaling=AutoscalingConfig(
            min_capacity=1,
            max_capacity=10,
            target_value=100,
            scale_in_cooldown=600,
            scale_out_cooldown=120,
        ),
        config_hash=config_hash,
    ),
)

How the assemble works

This part walks by means of the important thing implementation particulars of SagemakerAutoscalingConstruct.

Deregister

The Deregister customized useful resource calls deregister_scalable_target earlier than the endpoint replace. It handles ObjectNotFoundException for first deployments the place no scalable goal exists:

def handler(occasion, context):
    resource_id = occasion["ResourceProperties"]["ResourceId"]
    if occasion["RequestType"] in ("Create", "Replace"):
        attempt:
            shopper.deregister_scalable_target(
                ServiceNamespace="sagemaker",
                ResourceId=resource_id,
                ScalableDimension="sagemaker:variant:DesiredInstanceCount",
            )
        besides shopper.exceptions.ObjectNotFoundException:
            move  # No scalable goal on first deploy
    return {"PhysicalResourceId": f"deregister-{resource_id}"}

The dependency is established in CDK:

props.endpoint.node.add_dependency(deregister)

Register Scalable Goal

Regardless that CloudFormation treats the endpoint useful resource as full as soon as the replace is accepted, the scalable goal can solely be registered after the endpoint reaches InService. Moderately than block a single Lambda invocation on that wait, Register Scalable Goal makes use of the supplier’s asynchronous completion sample. The on_event handler returns instantly, and the supplier’s waiter re-invokes the is_complete handler on query_interval till the endpoint is InService. The supplier’s total_timeout (1 hour) bounds the wait, not the timeout of any single Lambda invocation:

def handler(occasion, context):
    props = occasion["ResourceProperties"]
    resource_id = props["ResourceId"]
    endpoint_name = resource_id.cut up("/")[1]
    if occasion["RequestType"] == "Delete":
        return {"IsComplete": True}
    standing = sm_client.describe_endpoint(EndpointName=endpoint_name)["EndpointStatus"]
    if standing == "Failed":
        increase Exception("Endpoint in Failed state")
    if standing != "InService":
        return {"IsComplete": False}
    as_client.register_scalable_target(
        ServiceNamespace="sagemaker",
        ResourceId=resource_id,
        ScalableDimension="sagemaker:variant:DesiredInstanceCount",
        MinCapacity=int(props["MinCapacity"]),
        MaxCapacity=int(props["MaxCapacity"]),
    )
    return {"IsComplete": True}

On stack deletion, the on_event handler deregisters the scalable goal to scrub up.

Connect Scaling Coverage

There’s a temporary window after RegisterScalableTarget returns the place the goal isn’t but queryable. Connect Scaling Coverage makes use of the identical asynchronous completion sample: the on_event handler returns instantly, and the is_complete handler stories completion solely as soon as the scalable goal is queryable, at which level it applies the coverage. The supplier’s total_timeout (quarter-hour) bounds the wait:

def handler(occasion, context):
    props = occasion["ResourceProperties"]
    resource_id = props["ResourceId"]
    if occasion["RequestType"] == "Delete":
        return {"IsComplete": True}
    resp = shopper.describe_scalable_targets(
        ServiceNamespace="sagemaker",
        ResourceIds=[resource_id],
        ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    )
    if not resp.get("ScalableTargets"):
        return {"IsComplete": False}
    shopper.put_scaling_policy(
        PolicyName=props["PolicyName"],
        ServiceNamespace="sagemaker",
        ResourceId=resource_id,
        ScalableDimension="sagemaker:variant:DesiredInstanceCount",
        PolicyType="TargetTrackingScaling",
        TargetTrackingScalingPolicyConfiguration=json.hundreds(props["ScalingPolicyConfig"]),
    )
    return {"IsComplete": True}

On stack deletion, the on_event handler deletes the scaling coverage to scrub up.

Cleansing up

To keep away from incurring future expenses, delete the assets:

uv run cdk destroy InferenceStack

This removes the SageMaker AI endpoint, endpoint configuration, mannequin, Lambda capabilities, IAM roles, Amazon CloudWatch alarms created by the goal monitoring scaling coverage, and deregisters the scalable goal.

Conclusion

On this submit, we confirmed you the right way to use a reusable CDK assemble to deregister and register auto scaling robotically to orchestrate SageMaker AI endpoint updates. The assemble makes use of three CloudFormation customized assets with specific dependency ordering and a ConfigHash property to maintain auto scaling intact throughout deployments.

To get began, clone the sample-sagemaker-endpoint-autoscaling repository and comply with the setup directions. The repository consists of an instance stack you’ll be able to deploy immediately, or you’ll be able to combine the assemble into your present CDK stack.

For extra details about Utility Auto Scaling for SageMaker AI, see Routinely scale Amazon SageMaker AI fashions. For extra details about CloudFormation customized assets, see Customized assets within the AWS CloudFormation Consumer Information.

Concerning the writer