Moving old attention state from GPU memory into CPU memory and SSD storage can support 73.02 times more concurrent model sessions per GPU and cut cost per session by 62.04 times, according to a simulation study of long-running chats, agents and document question answering. The capacity of the extra tiers—not a sophisticated placement algorithm—created most of the benefit.
Language-model servers store a key-value cache for earlier tokens so they do not recompute the full conversation at every step. That cache consumes scarce high-bandwidth GPU memory as sessions grow. Researchers modeled a hierarchy with relative capacities of one unit on the GPU, eight in CPU memory and 64 on SSD, calibrated with a learned execution-time predictor. They compared recency, reuse frequency, predicted reuse and a moving-average predictor, with and without prefetching.
At batch size one, decoding was compute-bound, so placement had little effect on token throughput. It mainly changed data transferred over PCIe and the delay before the first token. Recency created 2.3 times less migration traffic than reuse frequency for chats, while reuse frequency worked best for agents and document tasks. Prediction did not beat simpler rules, and even an oracle with future knowledge could not justify prefetch bandwidth. These are simulated results under specific hardware assumptions, but they suggest operators should buy useful tier capacity before optimizing speculative movement policies.