Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components | Amazon Web Services
When Salesforce set out to make Agentforce (Salesforce’s AI foundation for agents) highly available (HA) across multiple Availability Zones (AZs), the team faced a gap. Amazon SageMaker AI Inference Components (ICs) could cut GPU costs, but their default placement didn’t guarantee the Multi-AZ resilience Salesforce’s compliance bar required.
For Salesforce, the ICs delivered an 8x reduction in infrastructure costs by co-hosting multiple models on shared GPUs. However, this cost win introduced a new question: how do you make IC endpoints highly available across multiple AZs?
This post explores how Salesforce used the new IC Placement capability (surfaced through the SchedulingConfig parameter in the CreateInferenceComponent API) to meet their Multi-AZ HA compliance requirements.
The challenge: Single points of failure in IC deployments
By default, the SageMaker placement algorithm optimizes each IC deployment operation independently, distributing new copies evenly across instances without considering AZ balance. Even with a multi-AZ endpoint, this per-operation...
Copyright of this story solely belongs to amazon.com. To see the full text click HERE
