KV cache math: what long context costs, and what a 4-bit cache buys

Model weights are a fixed cost: you load them once and they don’t change. The KV cache grows with every token of context and with every parallel conversation. At long context it, not the weights, decides what fits on the card. Here’s the math, worked on the model I run every day.

What the KV cache is

During decode, every new token attends to every token before it. Recomputing the keys and values of the whole prefix at each step would make generation quadratic, so the engine computes each token’s key and value vectors once, in every attention layer, and keeps them. That store is the KV cache.

It’s per sequence. Two conversations running in parallel need two caches, even on the same model.

Bytes per token

kv_bytes_per_token = 2 × n_attn_layers × n_kv_heads × head_dim × bytes_per_value
                     └ K and V

Three things shrink it:

• Grouped-query attention. Several query heads share one key/value head, so n_kv_heads is a fraction of the query heads.
• Hybrid layers. Linear-attention layers, such as Gated DeltaNet, keep a fixed-size recurrent state instead of a per-token cache. Only full-attention layers count in n_attn_layers.
• Quantising the cache. This is the only one of the three you choose at serving time, not at training time.

Worked example: a 27B model at 262K

I don’t need the model config to get the number. The fork’s README reports 4,224 MiB of TBQ4 cache at 262,144 tokens, and TBQ4 stores 128 values in 66 bytes. That’s a 2-byte scale plus 64 bytes of 4-bit indices, so 4.125 bits per value.

4,224 MiB × 1,048,576 / 262,144 tokens  = 16,896 bytes per token
16,896 / 66 × 128                       = 32,768 cached values per token

Those 32,768 values are 2 × n_attn_layers × n_kv_heads × head_dim for this model. From there, every format is arithmetic:

KV format	Bits per value	Per token	8K	32K	128K	262K
FP16	16	64 KiB	512 MiB	2 GiB	8 GiB	16 GiB
Q8_0 (34 B / 32)	8.5	34 KiB	272 MiB	1.06 GiB	4.25 GiB	8.5 GiB
Q4_0 (18 B / 32)	4.5	18 KiB	144 MiB	576 MiB	2.25 GiB	4.5 GiB
TBQ4 (66 B / 128)	4.125	16.5 KiB	132 MiB	528 MiB	2.06 GiB	4.125 GiB
3-bit (50 B / 128)	3.125	12.5 KiB	100 MiB	400 MiB	1.56 GiB	3.125 GiB

The Q4_K_M weights take about 17 GB of a 24 GB card, which leaves roughly 7 GiB for the cache and the compute buffers. At FP16 that’s about 110K tokens before buffers; at Q8_0, about 210K. At 4 bits, 262K fits with room to spare.

Concurrency multiplies all of this. Four agents at 64K each need the same cache as one at 256K.

Why the cache is harder to quantise than the weights

Weights are quantised once, offline. You can calibrate on sample data, check perplexity and ship the file. The cache is written at runtime, one token at a time, by a kernel that has to be cheap. Nothing can be calibrated per input.

The values themselves are less friendly too. Keys have outlier channels: a few dimensions carry much larger magnitudes than the rest. That’s why KIVI quantises keys per channel and values per token. With one scale per block, the scale stretches to fit the outlier and every other value in the block loses resolution.

The two halves also fail differently. A key error shifts attention scores, and the softmax amplifies it. A value error shifts the output linearly.

My own tests on Qwen3.5-4B, all on CPU with the same prompt:

• 4-bit: 4-bit keys and values produced clean output.
• 3-bit keys and values: garbage, which I traced to the values.
• The cause: part of that was a packing bug in the 3-bit codec. It wrote a negative shift for bits at position 8 and above, which is undefined behaviour, and corrupted values 3–7 of every 8-value group.
• After the fix: 3-bit values produce clean text but are inherently lossier. Another TurboQuant implementation reports about +5.8% perplexity at 3.5 bits.

So 4-bit is the default in my fork. 3-bit values are refused unless you set LLAMA_ALLOW_TBQ3_KV=1.

How a 4-bit cache gets away with it: rotate first

TBQ4 runs each 128-value block through a randomised Walsh–Hadamard transform before quantising: a sign flip, the transform, another sign flip. The transform is orthonormal, so it moves no information. It spreads an outlier’s energy across all 128 coordinates, and what comes out looks close to Gaussian. A Gaussian is easy to quantise well: 16 Lloyd–Max centroids fitted to N(0, 1/√128) give a 4-bit index per value, plus one FP16 norm per block.

Orthonormality pays off a second time in attention. Dot products survive rotation: q · k = (Hq) · (Hk). So the upstream fused kernel never rotates the cache back:

• it rotates the query once;
• it reads keys and values directly as centroid × norm from raw 4-bit blocks, two table lookups per byte;
• it rotates the attention output back once at the end.

There’s no dequantise pass and no FP16 copy of the cache. On Metal, my port still dequantises to the original domain before attention. It runs at about 90% of Q4_0 speed, and the gap is that inverse transform.

Rules I use

1. Size the cache before you size the GPU. Bytes per token × context × concurrent sequences. The weights are the easy part of the budget.
2. Q8_0 is the safe default when it fits. Go to rotated 4-bit when context is the constraint. Treat 3-bit as an experiment, and re-run your evals when you try it.
3. Check quality on your own tasks, not on wiki-text perplexity. A cache that’s fine for chat can still break tool-call JSON.
4. Remember that the cache costs bandwidth, not just memory. Every decode step reads the whole cache for the attention layers. At 262K, a 4-bit cache is also a speed optimisation. The decode note covers that.

Sources: the block formats and quality findings are in the fork’s docs/kv-types.md. The fused rotated-domain kernel is upstream work from Indras-Mirror/llama.cpp-turboq-mtp. Per-channel key quantisation: Liu et al., KIVI (2024). Rotation before quantisation: Ashkboos et al., QuaRot (2024).