What Q4_K_M actually stores: weight quantisation from first principles

“Q4_K_M” is on every local-model download page, and most people pick it because everyone else does. Here’s what is actually inside the file, byte by byte, taken from the ggml source. And here’s why “4-bit” comes out at about 5 bits per weight on a real model.

One equation

Every scheme here stores a small integer q per weight, plus a few shared numbers per block of weights to turn it back into a float:

w ≈ d · q          symmetric  (Q4_0, Q8_0)
w ≈ d · q − m      with a minimum, so the grid can sit off-centre  (k-quants)

d is the step between levels. With b bits, a block’s range is cut into 2^b levels. Rounding to the nearest level leaves a uniformly distributed error, with a mean squared error of about d² / 12.

Each extra bit halves d, which cuts the squared error by 4×. Going from 4 bits to 3 quadruples it; going from 4 to 8 cuts it 256×. That’s the trade every format makes.

Why blocks

One scale for a whole tensor would let a single large weight set d for millions of others. So weights are quantised in small blocks, each with its own scale. The scale is overhead: a 16-bit scale per 32 weights costs 0.5 bits per weight. The simplest formats, straight from ggml-common.h:

typedef struct {
    ggml_half d;           // scale
    uint8_t qs[QK4_0 / 2]; // 32 × 4-bit
} block_q4_0;              // 18 bytes per 32 weights = 4.5 bits per weight

typedef struct {
    ggml_half d;           // scale
    int8_t  qs[QK8_0];     // 32 × 8-bit
} block_q8_0;              // 34 bytes per 32 weights = 8.5 bits per weight

So Q4_0 is a 4.5-bit format. The scale is what you pay for keeping outliers local.

K-quants: scales for the scales

The k-quants keep the small blocks but quantise the per-block numbers themselves, inside a 256-weight super-block:

typedef struct {
    ggml_half d;              // super-block scale for the sub-block scales
    ggml_half dmin;           // super-block scale for the sub-block minimums
    uint8_t scales[12];       // 8 scales + 8 mins, 6 bits each
    uint8_t qs[QK_K/2];       // 256 × 4-bit
} block_q4_K;                 // 144 bytes per 256 weights = 4.5 bits per weight

The 256 weights are 8 sub-blocks of 32. Each sub-block gets its own 6-bit scale and 6-bit minimum, and those are scaled by two FP16 numbers shared across the super-block:

w = d · scale_j · q  −  dmin · min_j          q ∈ [0, 15],  j = sub-block

The overhead is the same 0.5 bits per weight as Q4_0. But every 32 weights now get an asymmetric grid, a scale and an offset fitted to their own range, instead of a symmetric grid centred on zero. Same bits, better fit.

The rest of the family, all from the same header:

Type	Bytes per 256 weights	Bits per weight	Layout
Q3_K	110	3.4375	16 × 16, 6-bit scales
Q4_K	144	4.5	8 × 32, 6-bit scales and mins
Q5_K	176	5.5	Q4_K plus one high bit per weight
Q6_K	210	6.5625	16 × 16, 8-bit scales
Q8_0	272 (8 blocks of 34)	8.5	32-weight blocks, FP16 scale

The “_M”: not every tensor gets 4 bits

“Q4_K_M” is a recipe, not a type. The quantiser (src/llama-quant.cpp) picks a type per tensor:

• output.weight: Q6_K. It’s the projection that produces every token’s logits.
• attn_v and ffn_down: Q6_K in half of the layers. That’s the first eighth, the last eighth, and every third layer in between: i < n/8 || i ≥ 7n/8 || (i − n/8) % 3 == 2.
• Fused QKV tensors, where a model has them: Q5_K.
• Everything else: Q4_K.

Why those tensors? Under grouped-query attention, attn_v is several times smaller than attn_q. The source notes that in a 70B model it’s 8× smaller, so giving it more bits is nearly free. The layer pattern is a heuristic built into the quantiser: extra bits at both ends of the network, and on a regular stride through the middle.

The result: a 27B model at Q4_K_M is about 17 GB. That’s 17 × 8 / 27 ≈ 5.0 bits per weight on average, not 4.

What an outlier costs

A block’s step is set by its largest weight. Put one weight 10× larger than the rest into a 32-weight block and d grows 10×. The other 31 weights now carry 100× the squared error, because the error scales with d².

There are three defences, from cheap to expensive:

1. Small blocks, so an outlier only damages its own 32 weights. Every format above already does this.
2. An importance matrix. llama-imatrix records how strongly each input channel is activated on sample text. The quantiser then minimises error where it matters most, not uniformly. This matters most below 4 bits.
3. Rotation. Multiply by an orthonormal matrix, usually a randomised Hadamard, and the outlier’s energy spreads across the whole block. QuaRot and QuIP# do this to weights and activations so that both can go to 4 bits. It’s the same trick that makes a 4-bit KV cache work; the KV cache note covers it.

For weight-only GGUF files, the first two are usually enough. Rotation becomes necessary when you quantise what’s computed at runtime, such as activations or the KV cache, where there’s no offline pass to calibrate against.

Size and speed of a 27B model

The numbers below assume every tensor uses the same type. Real _M mixes land a little above. The bandwidth ceiling is from the decode note: one RTX 4090 at about 1,008 GB/s, one token per pass, weights only.

Type	Bits per weight	Weights	Decode ceiling
FP16	16	54.0 GB	18.7 passes/s
Q8_0	8.5	28.7 GB	35.1 passes/s
Q6_K	6.5625	22.1 GB	45.5 passes/s
Q5_K	5.5	18.6 GB	54.3 passes/s
Q4_K	4.5	15.2 GB	66.4 passes/s
Q3_K	3.4375	11.6 GB	86.9 passes/s

The RTX 4090’s 24 GiB is 25.8 GB. Whatever the weights leave is your cache budget, and the per-token cache sizes in the KV cache note turn it into a context length.

Fewer bits means fewer bytes per pass, so quantisation buys speed as well as memory. That’s why “it fits” and “it’s fast” usually point at the same format.

How I choose

1. Q4_K_M is the default for 20–35B models on a single 24 GB GPU. It’s the best quality per byte that still leaves room for a long-context cache.
2. Small models get more bits. They usually lose more to quantisation at the same bit width, and they’re small enough that Q5_K_M or Q6_K costs little.
3. Keep a Q8_0 build as the reference. Measure KL divergence against it (llama-perplexity --kl-divergence), then run your own task evals. Perplexity on generic text won’t tell you whether tool-call JSON still parses.
4. Below 4 bits, use an importance matrix and re-run the evals. That’s where formats stop being interchangeable.

Sources: block layouts and the Q4_K_M tensor rules are from ggml/src/ggml-common.h and src/llama-quant.cpp in llama.cpp, as of the source in my fork. Rotation-based quantisation: Ashkboos et al., QuaRot (2024); Tseng et al., QuIP# (2024).