Amazon SageMaker HyperPod now offers a Global Inference Gateway that routes model requests across multiple clusters and AWS regions through one managed endpoint. The service is meant to prevent requests from waiting on a saturated cluster while paid GPU capacity sits idle elsewhere.
Each participating cluster publishes load and health information. The gateway scores healthy clusters using signals that include capacity, error rate and regional proximity, then forwards the request over the AWS backbone. A cluster’s existing Inference Gateway chooses the final model pod. Operators can adjust the routing weights, while failed clusters are removed from service automatically.
Requests use AWS Signature Version 4 and IAM authorization, and cluster connections run through PrivateLink rather than the public internet. Customers must publish each cluster and then register it with a global gateway; AWS manages the routing fleet, private endpoints, certificates and health checks. Bagelcode said a beta routed traffic across two regions and continued serving when one cluster could not host the model. The feature reduces custom load-balancing work but remains tied to HyperPod and AWS networking.