Amazon SageMaker HyperPod has added model caching for inference, preloading container images and model weights onto cluster nodes before a serving pod needs them. The feature addresses a scaling delay that becomes severe when models contain hundreds of gigabytes of files.
Without a cache, a newly scheduled pod first pulls its inference image and then downloads weights from Amazon S3, FSx for Lustre or Hugging Face Hub. Amazon says a large serving image can take five to seven minutes, while a 145 GB model can add more than 20 minutes. A model above 600 GB, such as DeepSeek-R1, can leave an endpoint waiting 30 minutes or longer before it handles one request.
Caching moves those files to the node’s local NVMe storage in advance. Amazon reports local reads at roughly 7 GB per second and says prepared pods can typically begin serving in seconds rather than tens of minutes. That can make autoscaling more responsive and reduce repeated traffic to remote storage.
The improvement is not instantaneous model loading in every situation: a node needs enough local capacity, the required artifacts must already be cached, and the inference server still has startup work to perform. For predictable model sets, however, moving the largest transfers outside the request-time path removes a major source of cold-start delay.