SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints | Amazon Web Services
When you deploy a model to a GPU endpoint, the smallest unit of GPU you can request is one whole accelerator.
This constraint can cost you twice. First, you can’t host more models on an instance than it has accelerators. An ml.p4d.24xlarge has eight A100s, so it holds eight models, whether each one needs 40 GB or 14 GB. Second, and less obvious, it makes every deployment come down to a choice of which full GPU to deploy to. As your concurrency and prompt lengths grow, a full GPU on an older instance stops being enough, but a full GPU on a newer one is more than your model can use. You end up either under-served or paying for capability you never reach.
Today, Amazon SageMaker Inference supports GPU partitioning, which uses NVIDIA Multi-Instance GPU(MIG) to divide a physical GPU into independent partitions. Each partition receives a fixed share...
Copyright of this story solely belongs to aws.amazon.com. To see the full text click HERE
