QEvict lets long-context models recover evicted cache tokens

QEvict addresses the growing memory cost of the key-value cache, the stored attention state that lets autoregressive language models generate long outputs efficiently. Many cache-reduction methods evict tokens judged unimportant, but the paper argues that decision can be brittle.

As generation continues, token importance can drift. A token that looked safe to remove may later receive substantial attention under a full-cache model. QEvict keeps recoverable quantized states so evicted information can be restored when it becomes useful again.

The work is aimed at inference engineers pushing longer context windows under memory limits. It does not remove the trade-off between memory and quality, but it gives that trade-off a recovery path instead of making every eviction permanent.