Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod | Amazon Web Services
Large-scale model training is one of the most expensive workloads you can run on GPUs, and a single unhealthy accelerator can stall an entire distributed job. When training throughput drops, the hard part is rarely noticing that something is wrong. It is figuring out why and where. Teams often spend hours or days sifting through logs and disconnected dashboards to decide whether a slowdown came from a throttled GPU, a saturated network link, or a stalled data pipeline on the host.
Today, we are announcing enhanced observability in Amazon SageMaker HyperPod, a new capability that helps you diagnose GPU training issues faster. It brings together diagnostic and profiling capabilities for training workloads that run on both Amazon Elastic Kubernetes Service (Amazon EKS) and Slurm orchestrators:
Diagnostic summaryuses Amazon CloudWatch’s DetectGroupOutliers API to compare each GPU in a training job on utilization, memory, temperature, power, and SM clock, and flags...
Copyright of this story solely belongs to aws.amazon.com. To see the full text click HERE