AWS has demonstrated bidirectional key-value cache transfers in vLLM so multi-turn agents can reuse previously computed conversation context instead of processing it again on every request. In tests, the change improved time to first token by as much as 1.6 times for Qwen3-32B and 3.3 times for GLM-5-FP8.
Disaggregated inference uses separate nodes for prefill, which processes the prompt, and decode, which generates the response. Conventional systems send the cache only from prefill to decode. After a model replies, the decode node holds cached data for both the earlier prompt and the new tokens, but the prefill side cannot retrieve it. The next turn therefore repeats work, and a full cache miss can force the entire growing history through prefill again.
Sending cache data back allows prefill nodes to retain the context created during generation. That can reduce latency and accelerator use for assistants that repeatedly call tools and append results. The numbers come from AWS tests on EC2 P5en instances with Elastic Fabric Adapter, not a guarantee for every model or cluster. Benefits will vary with conversation length, cache hit rates, routing, and the cost of transferring state between nodes.