Long Context · Efficient AI · Language Models
KV Cache Compression Explained: Quantize, Offload, Compress or Replace
The KV cache, not the weights, runs out of memory at long context. Four 2026 routes attack it: 2-bit quantization for long decodes, lookahead offloading, 1:16 soft-token compression and fixed-size memory.
How it works
Every token a Transformer processes leaves a key and a value vector in every layer. Decoding the next token reads all of them, so the cache grows linearly with context and is re-read on every step. At 500K tokens the cache, not the model, is what no longer fits on the GPU, and at any length past a few thousand tokens decode is memory-bandwidth-bound, which means cache bytes are latency.
There are four places to intervene, and the papers in this explainer cover one each.
Store fewer bits per entry. KVarN quantizes keys and values to 2 bits without a calibration set. Its diagnosis is that prior methods were measured in a prefill-like setting, one shot of error, while reasoning models decode for thousands of tokens and re-read every quantized entry each step, so error compounds. The dominant cause is a few tokens with outlier scales. A Hadamard rotation spreads the outliers across channels, then a dual-scaling variance normalization along both axes of K and V fixes the per-token scales before rounding.
Keep fewer entries resident. FlashMemory-DeepSeek-V4 leaves the cache intact but predicts which chunks the next span of decoding will attend to. A dual-encoder Neural Memory Indexer scores chunks against the upcoming query state and keeps only the top ones on the GPU; the rest are offloaded. Because the indexer is a retriever, it can be trained with off-the-shelf retrieval tooling without loading the frontier backbone.
Compress before the decoder sees the tokens. End-to-End Context Compression at Scale trains a 0.6B encoder and a 4B decoder jointly so that each block of 4, 8 or 16 tokens becomes one soft token. The decoder’s prefill and cache shrink by that ratio. The encoder learns what the decoder needs, and a reconstruction auxiliary loss keeps the soft tokens from collapsing.
Change what is cached. Two papers replace per-head keys and values with something smaller. VideoMLA brings Multi-Head Latent Attention to causal video diffusion: a shared low-rank content latent plus one decoupled positional key, reconstructed into per-head K and V on the fly. delta-mem goes further and keeps the window fixed, carrying history in an 8 by 8 associative memory updated by a delta rule and read back as low-rank corrections to a frozen model’s attention.
Key numbers
| Route | Measurement | Value | Setting and source | Same harness? |
|---|---|---|---|---|
| Quantize | Precision that holds accuracy | 2-bit, calibration-free | KVarN on MATH500, AIME24, HumanEval | Yes, vs prior 2-bit methods |
| Quantize | Failure mode targeted | error accumulates over decode steps, driven by outlier token scales | KVarN analysis | n/a |
| Offload | Decode KV cache resident on GPU | 13.5% of full-context baseline on average | FlashMemory on LongBench-v2, LongMemEval, RULER | Yes |
| Offload | At 500K context | over 90% of cache overhead suppressed | FlashMemory | Yes |
| Offload | Accuracy change | +0.6 points average | FlashMemory | Yes |
| Compress | Compression ratios | 1:4, 1:8, 1:16 soft tokens | LCLM, 0.6B encoder + 4B decoder, 350B training tokens | Yes |
| Compress | Peak memory as context grows | plateaus; KV baselines run out near 512K to 1M | LCLM vs SnapKV, KVzip, Attention Matching | Yes |
| Compress | Accuracy | near the uncompressed 4B decoder at 1:4; measurable loss at 1:16 | LCLM on RULER, LongBench | Yes |
| Compress | Exact retrieval recovery | EXPAND(i) tool returning the original 512-token chunk approaches the uncompressed baseline | LCLM needle tests | Yes |
| Replace | Per-token KV memory | 224 scalars vs 3,072, a 13.7x cut (92.7%) | VideoMLA, every cached layer | Yes |
| Replace | Quality and throughput | VBench 0.859 at 60 s, best among evaluated; 1.23x throughput on one B200; up to 8x larger batches | VideoMLA | Yes |
| Replace | Fixed-size memory | 8 by 8 state; 1.10x over frozen backbone, 1.15x over best memory method, 1.31x on MemoryAgentBench | delta-mem | Yes |
The rows compare each method with its own baselines. No paper here runs quantization, offloading and upstream compression against each other on one model, so the table tells you what each route achieves, not which wins.
The trade each route makes
Quantization is the cheapest to adopt and the most bounded in upside: 2 bits against 16 is an 8x memory cut on the cache and no more, and the accuracy gap that remains at 2 bits is accepted, not eliminated. KVarN’s contribution is making 2-bit survive long decodes; it does not change the ceiling.
Offloading keeps full-attention semantics and reaches roughly 7x resident-cache reduction, but it adds a second model to serve and a prediction that can miss. FlashMemory’s +0.6 average hides the case where the indexer drops the one chunk with the answer, which aggregate accuracy will not show.
Upstream compression is the only route whose memory curve flattens with context, which is why it is the one that reaches 1M tokens on the same hardware where cache-eviction baselines run out. The price is exact recall: 1:16 costs measurable accuracy, and the fix the paper offers, an agent tool that expands original chunks on demand, moves cost back toward the raw tokens.
Replacing the cache is the most invasive and the most rewarding when it fits. VideoMLA’s 13.7x holds even though pretrained video attention is not low-rank, because the latent bottleneck sets the effective rank the model trains into. delta-mem’s 8 by 8 state is not a compressed cache at all; it is a different memory with a ceiling nobody has measured for hundreds of turns.
When to use which
- Reasoning models decoding long chains at 32K to 128K: KVarN-style 2-bit quantization measured under real decoding; make sure your kernel budget covers the Hadamard step.
- Single long documents or agent histories past 256K, exact answers needed: lookahead offloading. Keep an eye on needle-style failures the averages hide.
- Serving many long contexts where memory, not accuracy on every token, is the constraint: upstream soft-token compression at 1:4 to 1:8, with an expansion tool for the retrieval-heavy requests.
- Video or other streaming generation with a fixed window: latent KV as in VideoMLA; the batch-size headroom is the payoff.
- Conversation memory rather than document context: a fixed-size recurrent memory such as delta-mem, if the 1.15x over other memory methods matches your benchmark.
Limits and open questions
Evaluation surfaces differ per route: KVarN reports math and code; FlashMemory reports retrieval-friendly suites; LCLM uses 0.6B and 4B models; VideoMLA is one model on one B200; delta-mem reports relative multipliers whose absolute baselines you must read from the paper. Nothing here measures end-to-end tokens per second with all overheads on a shared server, and the interaction between routes, such as quantizing an offloaded cache or compressing a latent one, is unexplored. Model-side designs are moving the goalposts too: DeepSeek-V4’s compressed and heavily compressed attention layers report a KV cache at 10% of V3.2’s at 1M tokens, which is a different answer to the same problem.
FAQ
What is KV cache compression and why is it needed at long context?
The KV cache stores a key and value vector per token per layer and is re-read at every decode step, so it grows linearly with context and dominates GPU memory past a few hundred thousand tokens. Compression reduces its size by quantizing entries, offloading non-resident chunks, compressing tokens before the decoder sees them, or replacing per-head keys and values with a smaller latent.
Does 2-bit KV cache quantization hurt accuracy?
Less than it used to. KVarN shows the real damage comes from error accumulating over long decodes because of outlier token scales, and its Hadamard rotation plus dual-axis variance normalization sets a new 2-bit state of the art on MATH500, AIME24 and HumanEval measured under real decoding, without calibration data.
How much memory does KV cache offloading save compared with compression?
FlashMemory keeps 13.5% of the cache resident on average and cuts over 90% at 500K context with +0.6 accuracy, while keeping full attention. Latent context compression at 1:16 shrinks the decoder’s prefill and cache by that ratio and is the only route whose peak memory plateaus toward 1M tokens, at a measurable accuracy cost recoverable with an expansion tool.
Which KV cache compression technique should I use for a reasoning model?
For chains of thought at 32K to 128K, start with decode-tested 2-bit quantization such as KVarN. Past 256K, add lookahead offloading. Use upstream soft-token compression when many concurrent long contexts must fit in memory and exact recall on every token is not required.