Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Parts

When Salesforce got down to make Agentforce (Salesforce’s AI basis for brokers) extremely out there (HA) throughout a number of Availability Zones (AZs), the group confronted a niche. Amazon SageMaker AI Inference Parts (ICs) might minimize GPU prices, however their default placement didn’t assure the Multi-AZ resilience Salesforce’s compliance bar required.

For Salesforce, the ICs delivered an 8x discount in infrastructure prices by co-hosting a number of fashions on shared GPUs. Nonetheless, this value win launched a brand new query: how do you make IC endpoints extremely out there throughout a number of AZs?

This put up explores how Salesforce used the brand new IC Placement functionality (surfaced by the SchedulingConfig parameter within the CreateInferenceComponent API) to satisfy their Multi-AZ HA compliance necessities.

The problem: Single factors of failure in IC deployments

By default, the SageMaker placement algorithm optimizes every IC deployment operation independently, distributing new copies evenly throughout situations with out contemplating AZ stability. Even with a multi-AZ endpoint, this per-operation view means copies of a particular mannequin can find yourself inconsistently distributed throughout AZs, creating potential single factors of failure:

• Occasion-level failure: A single occasion crash takes down all copies of a mannequin.
• AZ-level failure: An AZ outage makes the complete mannequin unavailable.
• Compliance danger: Salesforce mandates 2-AZ assist for each manufacturing mannequin. Default placement for ICs, optimized for value alone, didn’t but meet their inner 2-AZ compliance bar.

The answer: SchedulingConfig

AWS launched the SchedulingConfig parameter within the CreateInferenceComponent API. It provides clients fine-grained management over IC copy placement throughout situations and AZs. Two key sub-parameters drive the HA conduct:

• AvailabilityZoneBalance: Controls cross-AZ distribution, balancing copies evenly throughout Availability Zones with configurable imbalance tolerance.
• PlacementStrategy (inside every AZ): SPREAD distributes copies throughout as many situations as potential for fault isolation. BINPACK packs copies onto fewer situations for utilization effectivity.

Code instance: Deploying an IC with Multi-AZ HA placement

Situation: Salesforce has a multi-AZ SageMaker endpoint with 4 situations distributed evenly throughout 2 Availability Zones (2 situations in AZ-1, 2 situations in AZ-2). The group needs to deploy a mannequin with 4 IC copies in order that they’re unfold throughout each AZs for top availability.

The next CreateInferenceComponent name deploys the mannequin with SPREAD placement and AZ balancing:

response = consumer.create_inference_component(
    InferenceComponentName="salesforce-llm-ic-ha",
    EndpointName="salesforce-multiaz-endpoint",
    VariantName="AllTraffic",
    Specification={
        'ModelName': 'salesforce-einstein-llm-v2',
        'ComputeResourceRequirements': {
            'NumberOfAcceleratorDevicesRequired': 1,
            'MinMemoryRequiredInMb': 65536
        },
        'DataCacheConfig': {'EnableCaching': True},
        'SchedulingConfig': {
            'PlacementStrategy': 'SPREAD',
            'AvailabilityZoneBalance': {
                'EnforcementMode': 'PERMISSIVE',
                'MaxImbalance': 1
            }
        }
    },
    RuntimeConfig={'CopyCount': 4}
)

With SPREAD, SageMaker distributes 4 copies throughout 4 situations: 2 in AZ-1 and a pair of in AZ-2. While you set MaxImbalance to 1, you configure the system to tolerate at most a 1-copy distinction between any two AZs.

For lighter fashions needing solely 2 copies, MaxImbalance: 0 enforces strict stability: precisely 1 copy per AZ:

# Lighter mannequin: strict 1-copy-per-AZ stability
'SchedulingConfig': {
    'PlacementStrategy': 'SPREAD',
    'AvailabilityZoneBalance': {
        'EnforcementMode': 'PERMISSIVE',
        'MaxImbalance': 0
    }
},
RuntimeConfig={'CopyCount': 2}

Scaling whereas preserving AZ stability

While you carry out scale-out and scale-in operations, SageMaker helps you preserve AZ stability by your configured SchedulingConfig parameters. While you scale out, SageMaker locations new copies to keep up even AZ distribution. While you cut back CopyCount, SageMaker symmetrically removes copies throughout AZs.

Necessary: By no means set CopyCount to 1 for HA-critical fashions. A single copy can solely reside in a single AZ, which implies you’d instantly break your 2-AZ compliance necessities.

update_response = consumer.update_inference_component(
    InferenceComponentName="salesforce-llm-ic-ha",
    RuntimeConfig={'CopyCount': 8}  # Scale out: 4 per AZ
)

Be aware: SchedulingConfig governs the location plan for every particular person scale operation. For ongoing consolidation and rebalancing over time (for instance, after repeated scale-in/scale-out cycles), configure the endpoint’s ScaleInPolicy with the CONSOLIDATION technique. With this configuration, a background sweeper periodically consolidates IC copies and releases idle situations whereas honoring AZ stability constraints.

# Step 1: Create an endpoint config with CONSOLIDATION ScaleInPolicy
consumer.create_endpoint_config(
    EndpointConfigName="salesforce-multiaz-endpoint-config-v2",
    ProductionVariants=[{
        'VariantName': 'AllTraffic',
        'InstanceType': 'ml.g5.xlarge',
        'InitialInstanceCount': 4,
        'ManagedInstanceScaling': {
            'Status': 'ENABLED',
            'MinInstanceCount': 2,
            'MaxInstanceCount': 8,
            'ScaleInPolicy': {
                'Strategy': 'CONSOLIDATION'
            }
        }
    }]
)

# Step 2: Replace the endpoint to make use of the brand new config
consumer.update_endpoint(
    EndpointName="salesforce-multiaz-endpoint",
    EndpointConfigName="salesforce-multiaz-endpoint-config-v2"
)

The three pillars of the location algorithm

The brand new placement algorithm launched three basic enhancements. Every instantly addressed Salesforce’s HA necessities:

1. Balanced ultimate distribution: The algorithm considers the stability of the ultimate distribution reasonably than solely fast placement wants.
2. Availability-aware distribution: SageMaker evenly distributes copies throughout AZs on a best-effort foundation. Endpoint and inference element replace operations persist multi-AZ placement, so HA is preserved throughout mannequin updates.
3. Inside-AZ optimization: Inside every AZ, the PlacementStrategy controls instance-level distribution. BINPACK packs copies onto fewer situations to maximise GPU utilization. SPREAD distributes copies throughout as many situations as potential for max fault isolation.

Salesforce selected SPREAD for Pillar 3, prioritizing fault isolation over packing density. This helps forestall a single occasion failure from taking down a number of copies of the identical mannequin.

Goal structure: Earlier than and after

Persevering with the previous situation: Salesforce’s endpoint has 4 situations throughout 2 AZs. Over time, the group deploys three ICs to this endpoint, every created in separate operations: IC1 (4 copies), IC2 (2 copies), and IC3 (2 copies).

Later, the group deploys IC3, a lighter mannequin needing solely 2 copies with strict AZ stability:

response = consumer.create_inference_component(
    InferenceComponentName="salesforce-light-model-ic3",
    EndpointName="salesforce-multiaz-endpoint",
    VariantName="AllTraffic",
    Specification={
        'ModelName': 'salesforce-summarizer-v1',
        'ComputeResourceRequirements': {
            'NumberOfAcceleratorDevicesRequired': 1,
            'MinMemoryRequiredInMb': 16384
        },
        'SchedulingConfig': {
            'PlacementStrategy': 'SPREAD',
            'AvailabilityZoneBalance': {
                'EnforcementMode': 'PERMISSIVE',
                'MaxImbalance': 0
            }
        }
    },
    RuntimeConfig={'CopyCount': 2}
)

With MaxImbalance: 0, you configure the algorithm to focus on precisely 1 copy per AZ, which helps you retain IC3 out there even when a whole AZ fails.

The next diagram illustrates how the default placement and the brand new SchedulingConfig placement differ when all three ICs are deployed to the identical endpoint:

Determine 1: Default placement in comparison with SchedulingConfig placement throughout two Availability Zones

Be aware: Fashions requiring a number of GPUs per copy (for instance, massive language fashions (LLMs) needing 4 accelerators) comply with the identical placement logic. SPREAD helps place every multi-GPU copy on a separate occasion, and AZ balancing distributes them evenly throughout zones.

Implementation concerns

The next sections cowl capability planning, configuration, and monitoring for Multi-AZ HA deployments.

Capability reservations

AWS strongly recommends On-Demand Capability Reservations (ODCR) for capability planning in AZ-constrained Areas. Salesforce pre-provisions reserved GPU capability in every goal AZ to assist confirm balanced IC placement. With out ODCR, on-demand capability constraints might restrict the distribution you need in high-demand Areas.

Be aware: The position algorithm helps partial deployment. If capability constraints forestall full AZ stability, SageMaker nonetheless locations copies on out there situations reasonably than failing the operation totally. This implies the function is usable even with out ODCR. Nonetheless, stability is probably not optimum.

Key configuration parameters

The next desk summarizes the really useful parameter values for Multi-AZ HA placement:

Be aware: DataCacheConfig and RoutingConfig are normal endpoint/IC configuration options unbiased of the IC placement technique. They’re included on this desk as a result of they complement HA deployments, however they aren’t a part of the SchedulingConfig placement function itself.

Monitoring AZ stability with SageMaker AI Insights

SageMaker AI Insights offers built-in observability for IC placement well being. With detailed observability enabled, the next metrics assist validate and preserve Multi-AZ HA:

• AZ skew (Reliability tab): Exhibits distribution imbalance share throughout your fleet. Use this to detect drift from balanced placement after scaling occasions.
• IC copy rely per AZ: Confirms every inference element maintains the anticipated copy distribution throughout Availability Zones.
• Rebalancing occasions and length: Tracks when SageMaker routinely rebalances copies and the way lengthy the operation takes.
• Inadequate Capability Error (ICE) rely per AZ: Screens ICE occasions by AZ and occasion kind. You should use this to assist decide if ODCR capability may have adjustment.

You’ll be able to entry these metrics within the SageMaker AI Insights dashboard and thru Amazon CloudWatch. An in depth walkthrough of observability for IC-based endpoints might be coated in an upcoming weblog put up.

The next screenshot exhibits an instance of the SageMaker AI Insights Reliability tab with AZ stability metrics:

Determine 2: SageMaker AI Insights Reliability tab with AZ stability metrics

Outcomes

By utilizing the IC Placement functionality, Salesforce’s AI group achieved:

• Multi-AZ HA compliance: Each mannequin deployment in Salesforce’s fleet satisfies their 2-AZ assist requirement.
• Eradicated single factors of failure: No mannequin could be absolutely taken offline by a single occasion or AZ failure.
• Preserved value effectivity: Multi-model co-hosting continues to ship infrastructure value financial savings, whereas SPREAD placement maximizes fault isolation throughout situations.
• Resilient scaling: Scale-up and scale-down operations protect multi-AZ distribution.
• Persistent HA by updates: Mannequin updates now not danger breaking AZ stability.

Key takeaways for enterprise AI groups

Salesforce’s journey to Multi-AZ HA with SageMaker Inference Parts provides a number of classes. Enterprise AI groups ought to contemplate the next:

1. Design HA on the IC stage, not simply the endpoint stage. Even with a multi-AZ endpoint, IC copies could be concentrated in a single AZ with out specific placement controls.
2. Use SchedulingConfig with SPREAD and AvailabilityZoneBalance for workloads with excessive availability necessities. That is the really useful beginning configuration for many fashions with HA necessities.
3. Pre-provision capability with ODCR. To realize balanced AZ placement, you could provision out there capability in every goal AZ. Don’t depend on on-demand capability for HA-critical deployments.
4. Set minimal CopyCount to 2 and minimal occasion rely to 2 because the HA baseline.
5. By no means scale to CopyCount: 1 for HA-critical fashions. A single copy can solely reside in a single AZ, which implies you’d instantly break your 2-AZ compliance necessities.
6. Monitor IC distribution constantly utilizing SageMaker AI Insights. Observe AZ Skew, IC Copy Depend per AZ, and Rebalancing Occasions on the Reliability tab to detect and remediate imbalance earlier than it turns into a reliability subject.

Conclusion

IC Placement provides enterprise AI groups the management they should meet strict availability necessities with out sacrificing value effectivity. For Salesforce, this functionality unlocked Multi-AZ HA compliance for his or her manufacturing Agentforce fashions. It additionally serves as a reference sample for enterprises working AI workloads with strict availability necessities on SageMaker.

AI workloads are more and more business-critical with strict uptime necessities. The power to regulate precisely how mannequin copies are distributed throughout infrastructure is now not a nice-to-have. It’s a basic requirement.

Additional studying

For extra info, see the next assets.

API references

Blogs and articles

Finest practices

In regards to the authors