AWS has added a push-based transfer mode for vLLM deployments that separate prompt processing from token generation, reporting up to a threefold improvement in time to first token and a 30% improvement in time per output token. The design targets large-model services that run the two inference phases on different engine instances.
The prompt, or prefill, phase creates a key-value cache containing intermediate attention data. A decode instance needs that cache before it can generate text. In the existing pull-based NIXL connector, the decode side waits for metadata, allocates destination blocks, prepares the transfer, and then reads the cache. Those sequential steps sit directly between prefill and generation.
The push design prepares and coordinates more of the transfer before it blocks output, moving data through supported remote-memory links such as AWS Elastic Fabric Adapter without staging it in host memory. AWS tested the approach on EC2 P5en instances. The gains are benchmark results for a specific disaggregated architecture, network, and workload; ordinary single-node vLLM deployments will not automatically see them. Operators must also weigh the added coordination against the scaling benefits of separating prefill and decode.