AWS has released a Ray LLM Deep Learning Container for serving one language model across multiple GPU nodes on Amazon EKS. Its reference deployment separates prompt processing, called prefill, from token generation, called decode, while presenting both stages through one OpenAI-compatible endpoint.
Prefill is compute-intensive because it processes the whole input, while decode repeatedly reads memory as it generates tokens. Moving them to separate nodes can prevent long prompts from delaying active responses and lets operators scale each phase independently. The trade-off is that the key-value cache produced during prefill must move quickly to the decode node.
The container bundles Ray Serve LLM, vLLM, Nvidia’s NIXL transfer library and AWS’s Elastic Fabric Adapter stack. AWS supplies a YAML-based builder that wires the two deployments together, reducing the version-matching and custom-image work normally required for distributed inference.
The sample uses two g6.8xlarge GPU nodes in one availability zone because EFA does not span zones. It is a technical reference, not a claim that two-node separation improves every workload. Teams still need to benchmark model size, prompt mix, network transfer and cost; small or lightly used models may be simpler and cheaper on a single node.