CUDA graphs in LLM decode: launch overhead, and why shape changes cost
A decode step is hundreds of small kernels, and launching each one costs CPU time. How CUDA graphs capture and replay the step, why a growing KV cache keeps invalidating them, and what raising the KV pad floor to 4096 changed in my fork.
At 82 tokens a second, a decode step has 12.2 ms. Most of that goes to streaming weights. Part of it, if you’re careless, goes to the CPU telling the GPU what to do next. CUDA graphs exist to remove that part, and how often they have to be rebuilt turns out to matter.
Death by a thousand launches
A decode step for a deep model is a long sequence of small kernels in every layer: norms, matrix multiplies, RoPE, attention, activations, residual adds. Each launch costs the CPU a few microseconds. A model with dozens of layers can easily issue around a thousand kernels per token. At about 4 µs each, that’s 4 ms, a third of the 12.2 ms budget, spent before counting CPU-side jitter.
That number depends on the model and the backend, so treat it as an order of magnitude. But the shape of the problem is real: decode kernels are short, so fixed per-launch costs are a large share of each one.
Capture once, replay many times
A CUDA graph records a sequence of kernel launches, with their arguments and dependencies, and replays the whole thing with a single call. The CPU cost of a step collapses from hundreds of launches to one, and the GPU can schedule the work back to back.
The constraint is that a captured graph is fixed. Same kernels, same shapes, same memory addresses. Some parameters can be updated in place, but if the topology or shapes change, the graph has to be captured again, and capturing isn’t free.
Why decode keeps breaking graphs
The KV cache grows by one token every step, so the attention kernels see a different sequence length every step. To keep graphs reusable, llama.cpp pads the KV view to a multiple of a fixed size and masks the unused tail. The shapes then change only when the context crosses a padding boundary, and only then does the graph need re-capturing.
Upstream hard-codes that padding at 256 tokens. A long agent session crosses a boundary every 256 tokens, and each crossing pays for a capture. In my fork I added a configurable floor, LLAMA_KV_PAD_MIN, defaulting to 4096. That’s 16× fewer re-captures than upstream.
The cost of a higher floor is more masked positions in the padded view. That matters most at short contexts and is noise at long ones.
Where the CPU still matters
Graphs shrink the CPU’s work but don’t remove it: sampling, scheduling, speculative-decoding bookkeeping and the server loop still run on one hot thread. On hybrid-core desktop CPUs, that thread landing on an efficiency core is visible in latency. My fork includes a proposal for pinning the hot thread to a performance core, written for an i9-12900K.
How to see it
- Profile a decode run with Nsight Systems (
nsys profile). Gaps between kernels on the GPU timeline are CPU overhead you’re paying for. - Compare with graphs off. llama.cpp’s CUDA backend honours
GGML_CUDA_DISABLE_GRAPHS=1, which gives a quick A/B of how much the graphs save on your model. - Watch re-captures over a long session. If tokens per second dips at regular intervals, padding boundaries are a suspect.
Rules I use
- Measure launch overhead before optimising kernels. On small or heavily quantised models it can be a bigger slice than any single kernel.
- Keep shapes stable. Padding, fixed batch sizes and preallocated buffers all exist to keep graphs valid.
- Give the hot thread a fast core. It’s the cheapest latency fix on a desktop box.
Sources: NVIDIA’s CUDA Graphs programming guide. The KV pad floor and the P-core proposal are in my fork. The launch-overhead figures are an illustrative estimate, not a profile of a specific model.