AWS has released an Agent Toolkit skill that evaluates and configures distributed key-value caching for self-hosted language models. A KV cache stores the attention state created while a model reads a prompt; sharing that state across GPU servers can avoid recomputing long conversation histories when a request lands on a different machine.
The skill works with coding agents including Kiro, Claude Code and Codex. It inspects a GPU fleet, asks for missing workload details, and uses AWS’s KVTrace tool to estimate cache hits and costs before creating infrastructure. It can then add an LMCache connector to vLLM or SGLang and align cache identifiers and data types across servers. High-impact changes still require explicit approval.
AWS reports that suitable deployments can cut inference cost by up to 40% and time to first token by up to 80%, but short prompts may be faster to recompute than retrieve. The documented setup also requires specific software versions, EFA-capable GPU instances and large ElastiCache for Valkey nodes. Multi-tenant systems need isolation or tenant-specific hashes because response timing could otherwise reveal whether another customer submitted the same text.