Decode is memory-bound: the ~59 tok/s ceiling on an RTX 4090, and how speculative decoding gets past it

My box decodes at up to 82 tok/s. On paper, one RTX 4090 can’t run this model faster than about 59 forward passes a second. Both numbers are right, and the gap between them is speculative decoding. Here’s the math behind both.

The ceiling

A decode step on a dense model reads every weight once, to produce one token for one sequence. Each weight read does about two floating-point operations, a multiply and an add. The GPU can do dozens of times more arithmetic per byte than that, so the step is limited by how fast the weights stream out of VRAM:

RTX 4090 memory bandwidth          ≈ 1,008 GB/s
27B model at Q4_K_M                ≈ 17 GB of weights
ceiling, one token per pass        ≈ 1,008 / 17 ≈ 59 passes/s

Compute isn’t close to the limit. 27B parameters × 2 FLOPs is 54 GFLOP per token; at 59 tokens a second that’s about 3 TFLOP/s, a few percent of what the card can do. Decode spends its time waiting on memory.

At long context, the KV cache joins the weights on every pass, because attention reads the whole cache for every new token. At 262K with a 4-bit TBQ4 cache, that’s another 4.4 GB per pass:

weights + full 262K TBQ4 cache   ≈ 17 + 4.4 = 21.4 GB per pass
ceiling                          ≈ 1,008 / 21.4 ≈ 47 passes/s

At FP16 the cache would be 17 GB, as large as the weights, and the ceiling would drop to about 29 passes a second. A smaller cache is a speed optimisation, not just a memory one. (The KV cache note has the sizes.)

That gives two levers: read fewer bytes per pass (quantisation), or get more than one token out of each pass.

Speculative decoding

A cheap drafter proposes k tokens. The large model then scores all of them in a single forward pass. Because the pass is memory-bound, checking k tokens costs about the same as generating one. The engine keeps the longest prefix the large model agrees with. At the first disagreement it uses the large model’s own token instead, so every pass yields at least one token.

With sampling, “agrees” means rejection sampling: a draft token x is accepted with probability min(1, p(x) / q(x)), where p is the large model’s distribution and q the drafter’s. That keeps the output distribution exactly the large model’s. Speculation changes the speed, not the text you’d get.

If each draft token is accepted with probability α, independently, the expected number of tokens per pass is (Leviathan et al., 2023):

E[tokens per pass] = (1 − α^(k+1)) / (1 − α)

Acceptance α	k = 1	k = 3	k = 5
0.35	1.35	1.52	1.54
0.5	1.50	1.88	1.97
0.6	1.60	2.18	2.38
0.7	1.70	2.53	2.94
0.8	1.80	2.95	3.69
0.9	1.90	3.44	4.69

Read the table by rows:

• High acceptance rewards deep drafts: at 0.9, five drafts give 4.7 tokens per pass.
• Low acceptance makes depth almost worthless: at 0.35, going from three drafts to five adds 0.02 tokens per pass, while every extra draft still costs compute and widens the verification batch.

The full speedup also counts the drafter’s cost. If one draft step costs c of a large-model pass:

speedup = (1 − α^(k+1)) / ((1 − α) · (k·c + 1))

That’s why a cheap drafter matters as much as a good one.

MTP: drafting without a draft model

A separate draft model needs its own VRAM and its own KV cache. Multi-token prediction (MTP) heads avoid that. They’re small extra layers, trained together with the model, that predict tokens t+1, t+2… from the trunk’s own hidden state. DeepSeek-V3 popularised them; Qwen3.5 and 3.6 ship with them. A draft step is a fraction of a pass, so c is small.

The catch is that draft quality is capped by the heads. You can’t swap in a better drafter without retraining them, so acceptance is a property of the model you picked as much as of the engine.

A consistency check on my own numbers

82 tok/s against a ceiling of about 59 passes a second means at least 1.4 tokens per pass, even if every pass ran at the theoretical limit. Real passes are slower than the ceiling, so the true figure is higher. At 1.8–2 tokens per pass with three drafts, the formula puts acceptance at roughly 0.47–0.54. That sits inside the 50–68% range the fork’s README records for this model’s heads.

This is back-of-envelope, not a profile, but the numbers hang together. Most of the remaining headroom is on the model side, in retraining the MTP heads, not in the kernel.

Two things the math says to do

Adapt the depth to the acceptance. The table shows extra drafts are wasted when acceptance drops, and acceptance does drop on hard stretches of text. My fork’s --spec-adapt throttles draft depth when measured acceptance falls below 35%, the row where extra drafts buy almost nothing.

Match the draft sampler to the target’s. Expected acceptance is Σ min(p(x), q(x)) over all tokens, the overlap between the two distributions. A greedy drafter puts all its mass on one token, so acceptance collapses to p(argmax). Against a target sampling at temperature 0.6, that’s often well below what a drafter sampling from a similar distribution would get. That’s what switching my fork’s MTP draft sampler from greedy to stochastic changes. Separately, the production sampler settings (temperature 0.6, top-p 0.95, min-p 0.05) took decode from 62 to 82 tok/s and stopped tool-call loops.

Rules I use

1. Compute the ceiling first: bandwidth ÷ bytes per pass. If you’re near it, kernel tuning won’t save you. Fewer bytes or more tokens per pass will.
2. Batching buys throughput, not latency. N sequences share one read of the weights, so a server’s total tokens per second can multiply. A single user’s stream is still bound by the ceiling above.
3. Pick draft depth from measured acceptance, and keep measuring. It changes with the text, the sampler and the model.
4. Long context makes everything worse. The cache read grows with context. Quantise the cache before you buy more bandwidth.

Sources: Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding (ICML 2023) for the acceptance and speedup formulas. RTX 4090 bandwidth is NVIDIA’s spec (21 Gbps GDDR6X on a 384-bit bus). The engine numbers and the --spec-adapt threshold are from my fork.