SageMaker HyperPod can now health-check your GPU cluster proactively, even while it's running!

Hey there, it's Shiichan! Today I've got news that makes running GPU clusters a bit more reassuring, especially if you're doing large-scale ML training.

AWS What's New aws.amazon.com

What was announced?

According to AWS's What's New, Amazon SageMaker HyperPod's deep health checks now support Slurm-orchestrated clusters built with continuous provisioning. This means you can now proactively verify GPU accelerator health on running instances, at any time.

The story so far

Continuous provisioning lets you scale instance groups asynchronously, so you can start training right away instead of waiting for an all-or-nothing setup where every instance has to be ready at once. But until now, there was no way to pair that asynchronous flexibility with comprehensive hardware validation for the worker nodes streaming in. Even a single unhealthy GPU node could waste hours of compute time and delay critical workloads.

What changes

You can now target entire instance groups or individual instances and run comprehensive hardware stress tests and connectivity tests before committing compute resources to a job. Because continuous provisioning adds worker nodes to the cluster asynchronously, you can run deep health checks on each new node as it comes online, validating hardware before scheduling jobs on it, all without interrupting workloads already running on healthy nodes.

Dive Deep

Instances undergoing checks are automatically isolated from workload scheduling, and they're automatically returned to service once they pass. Progress and results are visible at both the instance-group and instance level through the SageMaker console and APIs, giving you clear visibility into GPU health, network connectivity, and multi-node communication performance.

It's also paired with HyperPod's automatic node recovery capability, so instances that fail the check get automatically rebooted or replaced. This is available across every AWS Region where SageMaker HyperPod operates. For setup details, check the SageMaker HyperPod user guide.

Wrap-up

• SageMaker HyperPod's deep health checks now support Slurm clusters using continuous provisioning
• You can run hardware stress tests and connectivity tests on running instance groups or instances, any time
• New nodes joining asynchronously get validated without interrupting existing workloads
• Instances under check are automatically pulled from scheduling and returned to service once they pass
• GPU health, network connectivity, and multi-node communication performance are visible via console and API
• Paired with HyperPod's automatic node recovery, failed instances get rebooted or replaced automatically

This update is a great fit for infrastructure and ML engineers running large-scale GPU clusters for training!