Amazon has launched a GPU-aware request router for AI models running on SageMaker HyperPod and Amazon EKS. The Inference Gateway replaces blind round-robin balancing with live signals from each serving pod, aiming to keep requests away from overloaded GPUs and make better use of existing capacity.
Its scheduler weighs key-value cache usage, queue depth, active requests, cached prompt prefixes and whether a requested LoRA adapter is already loaded. LoRA adapters are small model customizations; routing to a GPU that already holds one avoids a costly swap. The gateway accepts OpenAI-compatible requests and discovers model pools through Kubernetes configuration, so existing clients and servers do not require code changes.
In Amazon’s benchmarks across four models from 8 billion to 235 billion parameters, first-token latency improved by as much as 82 percent compared with Kubernetes round-robin routing. One cited case fell from 4.4 seconds to under 800 milliseconds. Benefits were largest with mixed hardware, bursty demand and shared prompt prefixes; uniform replicas under steady traffic performed about the same as round robin. Those figures come from AWS tests using the default configuration and should be validated on each workload. Per-cluster routing is available now in regions supporting the inference add-on, while cross-cluster and regional coordination is planned as a second tier.