KVarN: 2-Bit KV-Cache Quantization Without Calibration

Quick answer

KVarN is a calibration-free KV-cache quantizer that holds accuracy at 2-bit precision where prior methods collapse. Its core insight: in autoregressive decoding, quantization error does not stay flat — it compounds step after step, mostly because a few tokens get the wrong scale. KVarN applies a Hadamard rotation followed by a dual-scaling variance normalization across both axes of the key and value matrices, which flattens those outlier token scales and sets a new state-of-the-art for 2-bit KV-cache quantization on MATH500, AIME24, and HumanEval.

Why decoding breaks KV-cache quantization

Most KV-cache quantization papers are measured wrong, and KVarN’s first contribution is pointing this out. Existing methods are evaluated in a prefill-like setting — quantize a fixed context, read one answer — where a fixed amount of error is tolerable. But test-time scaling, the trick behind modern reasoning models, makes the model decode for thousands of tokens, and the KV-cache grows the whole time. That is the regime that actually matters, and it behaves differently: each quantized key/value gets attended to again and again, so a small error at step 50 keeps influencing every later step. The authors show this empirically — error accumulates across timesteps rather than staying bounded.

What actually causes the drift

The dominant culprit is incorrect token scales, not random rounding noise. A handful of tokens carry outlier magnitudes; when you squeeze everything into 2 bits with a single shared scale, those outliers either saturate or force the scale so wide that ordinary tokens lose all their resolution. Either way the per-token representation is distorted, and because attention re-reads those tokens throughout decoding, the distortion gets amplified instead of averaging out. This reframes the problem from “make rounding finer” to “fix the scales before you round.”

How KVarN works

KVarN does two things, in order, with no calibration set required. First a Hadamard rotation mixes the channels so that outlier energy is spread across dimensions instead of concentrated in a few — a now-standard trick for taming activation outliers, here applied to K and V. Then dual-scaling variance normalization normalizes variance along both axes of the K and V matrices rather than one. The second axis is the part competitors miss: scaling only per-channel (or only per-token) leaves the orthogonal direction’s outliers intact, and those are exactly the token-scale errors that drive accumulation. Being calibration-free matters in practice — there is no dataset to collect, no fitting step, and nothing that can silently mismatch the deployment distribution.

Key results

• KVarN sets a new state-of-the-art for KV-cache quantization at 2-bit precision on generative benchmarks, measured under real autoregressive decoding rather than prefill.
• It reports gains on MATH500, AIME24, and HumanEval — i.e. math reasoning and code generation, the workloads where long decode chains and test-time scaling matter most.
• The headline is the reduction in error accumulation over existing baselines: the gap widens as generation gets longer, which is precisely where prefill-tuned quantizers degrade.
• A vLLM implementation is released, so the method targets a production inference stack, not just an offline study.

Why this matters now

Reasoning models are memory-bottlenecked exactly because they think out loud for a long time. The KV-cache, not the weights, is what blows up during long decodes, and 2-bit is roughly the floor where you get a meaningful memory win. A calibration-free method that survives at 2 bits during decoding lets you fit longer chains or larger batches on the same GPU without a tuning ritual per model. The honest framing here is the contribution: KVarN is less about a new clever transform and more about correctly diagnosing that decode-time error compounds, then targeting the token-scale cause directly.

Limits and open questions

The paper’s evaluation is concentrated on verifiable reasoning and coding benchmarks; how 2-bit KVarN behaves on long open-ended generation, retrieval-heavy contexts, or multilingual decoding is not the focus, and those can stress quantization differently. The Hadamard-plus-dual-normalization pipeline adds compute and a non-trivial kernel to every decode step, so the relevant question for practitioners is end-to-end latency and throughput, not just accuracy at a bit-width — the abstract leads with quality, not speed. And 2 bits is aggressive: “state-of-the-art at 2-bit” still means living with whatever accuracy gap remains versus a 16-bit cache, which only some applications will accept. Independent reproduction across model families would firm up how general the dual-axis fix is.

FAQ

What problem does KVarN solve that other KV-cache quantizers miss?

KVarN targets error accumulation during autoregressive decoding. Prior quantizers are tuned and evaluated in prefill-like settings where error stays bounded; KVarN shows that under long decoding, quantization error compounds across timesteps, driven mostly by wrong token scales, and fixes that specific cause.

How does KVarN achieve 2-bit KV-cache quantization without calibration?

It applies a Hadamard rotation to spread outlier energy across channels, then dual-scaling variance normalization across both axes of the K and V matrices. Because normalization is computed from the matrices themselves, no calibration dataset or fitting step is needed.

Where does KVarN show the biggest gains?

On generative reasoning and code benchmarks — MATH500, AIME24, and HumanEval — at 2-bit precision, where it sets a new state-of-the-art and where the advantage grows as the generated sequence gets longer.

Can I use KVarN in production?

The authors release a vLLM implementation, so it is aimed at a real inference stack. The open practical question is the added per-step compute from the rotation and normalization, which affects throughput and latency.

One line: diagnose that decode-time KV-cache error compounds via bad token scales, then fix the scales — and 2-bit caches survive long reasoning chains. Read the original paper on arXiv.