Reduce inference cold starts on Amazon SageMaker HyperPod with model caching | Amazon Web Services
When you deploy a large language model (LLM) for inference on Amazon SageMaker HyperPod, there’s a gap between when you request a pod and when it’s ready to serve traffic. This gap is dominated by two sequential downloads: the inference server container image from Amazon Elastic Container Registry (Amazon ECR), and the model weights from your storage source, which can be Amazon Simple Storage Service (Amazon S3), Amazon FSx for Lustre, or HuggingFace Hub. For smaller models, this might be a few minutes. For large models, like DeepSeek-R1 at 600+ GB, you’re looking at 30 minutes or more before a single request can be served. Every scale-out event goes through the same download cycle, which means your autoscaling response time is gated by network throughput to your storage backend.
Today we’re launching model caching for Amazon SageMaker Inference on HyperPod. Model caching pre-loads model weights and...
Copyright of this story solely belongs to aws.amazon.com. To see the full text click HERE
