Amazon SageMaker HyperPod now includes enhanced observability intended to identify why large distributed training jobs slow down. A diagnostic summary compares GPUs within the same job and points to the affected cluster, node, accelerator and process rank instead of leaving engineers to reconcile separate dashboards.

CloudWatch’s group-outlier detection compares utilization, memory, temperature, power and clock speed. Static rules look for GPU XID and memory errors, network throughput drops, retransmissions, thermal throttling and overheating. The feature supports clusters orchestrated by both Amazon EKS and Slurm.

Basic cluster, host, GPU and job metrics are collected every 30 seconds. AWS estimates that collection uses 1–2% CPU and 250–300 MB of memory. Advanced accelerator, host and Elastic Fabric Adapter telemetry is optional because sending more measurements to CloudWatch or managed Prometheus increases monitoring costs.

CloudWatch is now the default metric store, while teams standardized on Prometheus and Grafana can keep those tools. Enhanced observability is available in every region that offers HyperPod. The dashboards can shorten the search for a failing component, but they do not prevent application bugs or guarantee that every low-throughput job is caused by infrastructure.