A new preprint describes KVBoost, a cache layer designed to avoid recomputing repeated prompt content when that content does not appear at the beginning of every request. Existing prefix caches work best when prompts share the same contiguous opening; KVBoost stores and matches smaller chunks wherever they occur.
The system separates a chunk’s content identity from its position using two hashes. Because independently cached chunks can create errors where attention crosses boundaries, it selectively re-encodes boundary regions or runs a probe to find high-deviation tokens that need recomputation. It also supports 8-bit and 4-bit cache compression and evicts chunks according to estimated importance under a fixed memory budget.
On 1,000 bug-localization samples using Qwen2.5-3B, the authors report that KVBoost cut time to first token from 639.1 milliseconds to 142.4 milliseconds. That is a 4.49-fold reduction and 16 percent better than prefix caching, while measured accuracy remained effectively unchanged at 99.2 versus 99.1 percent.
The evidence is limited to one small model and one benchmark. KVBoost is presented as compatible with Hugging Face decoder models that use rotary positional embeddings, but broader hardware and workload tests are still needed.