262K context on one RTX 4090: what I added to llama.cpp-turboq
The engine already kept a 4.125-bit KV cache inside flash attention. Running coding agents on it all day took async checkpoints, fewer CUDA graph re-captures, a stack of MTP fixes and a sampler tune worth +31%.
My local inference box is one RTX 4090 with 24 GiB of VRAM. It serves a 27B model with a 262,144-token context to my agent harnesses, all day. This note is about what it took to make that boring.
The setup
- GPU: one RTX 4090, 24 GiB, every layer offloaded.
- Model: Qwopus3.6-27B-v2-MTP, a Qwen3.6 27B fine-tune with multi-token-prediction heads, Q4_K_M.
- Context: 262,144 tokens, with a 4-bit TBQ4 KV cache.
The production server runs with:
--temp 0.6 --top-p 0.95 --min-p 0.05 # fixes agent tool-call loops, decode 62 → 82 tok/s
--reasoning on --reasoning-budget 8192
--jinja
--spec-type mtp --spec-draft-n-max 3 # MTP speculative decoding
--spec-adapt # adaptive draft-depth guard
-ctk tbq4_0 -ctv tbq4_0 # 4.125 bpv KV, 4.1 GiB at 262K
--flash-attn on
-c 262144 -ngl 99What I started from
My repo is a fork of Indras-Mirror/llama.cpp-turboq-mtp, which is itself a fork of llama.cpp. The headline feature there is theirs, not mine: a flash-attention kernel that reads raw TBQ4 blocks straight from global memory and dequantises them inside the inner loop. TBQ4 stores 128 values in 66 bytes: a 2-byte scale plus 64 bytes of 4-bit indices, which is 4.125 bits per value. Attention runs in the rotated (Hadamard) domain, so there’s no separate dequant pass and no F16 buffer.
That kernel is why the context fits at all. At 262K tokens the TBQ4 cache takes about 4.1 GiB. An FP16 cache would take 16 GiB, two thirds of the card, before a single weight is loaded.
So I didn’t write the kernel. My work starts where a benchmark run ends: when a coding agent starts hitting the server for hours, with long prompts, tool calls and context near the limit.
1. Checkpoints were stalling prefill
The server takes prompt checkpoints during prefill, so a follow-up request can reuse the cache instead of reprocessing the whole prompt. During a 262K prefill that means 32 state readbacks of about 149 MiB each. Every one did a full ctx->synchronize() followed by a blocking ggml_backend_tensor_get per state tensor. The compute stream sat idle while the host copied memory.
I made the capture asynchronous:
create_checkpoint (prefill loop, no stall)
└─ async D2H into a pinned staging slot (cudaStreamPerThread)
└─ cudaLaunchHostFunc: staging → pageable store, release the slot
follow-up request (GPU idle)
└─ if a copy is still in flight: cudaStreamSynchronize // cheap, GPU idle
└─ restore the checkpointThe staging ring has two slots, so pinned host memory never goes above about 300 MiB, however many checkpoints a request takes. Pinning all 32 would need about 4.7 GB, and that host doesn’t have it to spare. The data only gets synchronised when a checkpoint is actually consumed, and by then the GPU is idle anyway.
2. Fewer CUDA graph re-captures
Upstream pads the KV cache to a hard-coded 256. I added a configurable floor, LLAMA_KV_PAD_MIN, defaulting to 4096. That’s 16× fewer CUDA graph re-captures than upstream.
3. Agents looping, and +31% from the sampler
Under agent workloads the model kept falling into tool-call loops. Three changes stopped that:
- Sampler:
--temp 0.6 --top-p 0.95 --min-p 0.05stopped the loops and raised decode from 62 to 82 tok/s, which is +31%. - Stochastic draft sampler: the MTP draft sampler no longer samples greedily.
- Adaptive breaker: a single-sample breaker catches tool-call loops that still start.
--spec-adapt throttles MTP draft depth when acceptance drops below 35%, so a bad stretch of text doesn’t burn compute on drafts that get rejected anyway.
82 tok/s is peak decode at stable generation. It’s not the setting I’d run all day.
4. Long-session correctness
None of these show up in a short benchmark run. They show up in long agent sessions:
- MTP shadow context: fixed its synchronisation with the main context, and a data race between it and the async checkpoints.
- Recurrent-state drift: fixed drift in Gated DeltaNet at high context that ended in a dead loop.
- Out-of-bounds crash: fixed an MTP tensor crash, and made the server handle a full context gracefully instead of failing.
- M-RoPE rollback: MTP rejection rollback now works for models like Qwen3.8-27B, with a hardened 1D path.
- Loop regression: the pending MTP stash re-syncs whenever the target’s memory changes.
- Prefill speed: a fused chunk-pipeline kernel for DeltaNet prefill.
5. Serving it to harnesses
The harness on the other side expects an OpenAI-compatible API and doesn’t care what’s behind it, so the server has to behave:
- Hot-swap: models swap in-process through a preset registry, with no separate router.
- Reasoning effort:
reasoning_effortaccepts a word or a raw token budget, and a request overrides the server config. - Prompt trimming: trimming is message-aware at parse time, and it never drops the last message.
6. The same cache on Apple silicon
I ported TBQ3 and TBQ4 KV cache support to Metal, fixing a 3-bit packing bug on the way. I also added rotated-domain dequant for attention, so the Metal path skips a separate dequant pass too.
The numbers
| What | Value |
|---|---|
| Context | 262,144 tokens on 24 GiB |
| KV cache | TBQ4, 4.125 bpv, 4.1 GiB at 262K |
| Decode, peak stable | 82 tok/s |
| Sampler tune | 62 → 82 tok/s (+31%) |
| CUDA graph re-captures | 16× fewer |
| Pinned RAM for checkpoints | ≤ ~300 MiB instead of ~4.7 GB |
What’s next
MTP acceptance is now limited by the draft head, not the engine. The next lever is on the model side: retraining the MTP head with a FastMTP objective. The repo has the write-up.
The code, commit by commit: github.com/jtrefon/llama.cpp-turboq-mtp.