A single failed GPU can stop an expensive distributed training job even when every other worker remains healthy. AWS has demonstrated how Nvidia’s Resiliency Extension, or NVRx, can restart PyTorch workloads in seconds and write checkpoints without pausing all workers.
The setup ran Fully Sharded Data Parallel training on Amazon EKS clusters containing two to eight nodes, each with eight H100 GPUs. Asynchronous checkpointing sends each worker’s state to a background process and lets training continue while data is written. In the tested synchronous setup, checkpoint input and output had consumed as much as 40% of total runtime.
Two restart layers cover different failures. An in-process wrapper can abandon a broken communication group, test the remaining hardware and resume from the latest checkpoint without restarting Python. A launcher monitors heartbeats and creates fresh worker processes after harder failures such as an out-of-memory kill or operating-system hang. Node loss still requires the cluster orchestrator.
AWS reports training efficiency above 99% and recovery measured in seconds in its H100 benchmarks. Those figures depend on its network and shared Lustre storage configuration, and loading a large checkpoint can dominate recovery at scale. Teams can adopt the checkpoint and restart components separately, matching the added complexity to the failures they actually encounter.