Trefon.
L2 Inference / optimization

The same model, faster and cheaper.

Most inference problems are not model problems. They are memory, batching, cache and scheduling problems. I profile the stack you run, change the part that is actually limiting it, and show the before and after on your workload.

Describe your problem See the work write-up

+31%

decode after a production tune

262K

context on one 24 GB card

16×

fewer CUDA graph re-captures

§01 the symptom

How this usually shows up.

Throughput plateaus.

One request is fine. Ten concurrent requests and the queue never drains. The GPU is busy, but not with useful work.

Context eats the card.

Long prompts only fit at short context, or the server falls back to CPU and latency collapses. The KV cache, not the weights, is the limit.

The bill scales with success.

Every new customer adds tokens. The serving bill grows faster than revenue, and the price cut you need is in the stack, not the contract.

§02 what I change

Where the time and memory go.

  1. 01

    Measure first

    A load test on your prompts, your context lengths and your concurrency. I find the actual bottleneck before changing anything: prefill, decode, KV-cache bandwidth, scheduling or the host.

  2. 02

    KV cache

    The cache decides how much context fits and how fast decode runs. Quantised caches, paged allocation, prefix reuse: I pick the combination your accuracy budget allows.

  3. 03

    Quantisation

    Weights and cache quantised to the lowest precision that still passes your evals. Measured on your tasks, not on a leaderboard.

  4. 04

    Scheduling and batching

    Continuous batching, chunked prefill, speculative decoding, CUDA graphs. The goal is fewer idle gaps and more tokens per second per card.

  5. 05

    Serving surface

    An OpenAI-compatible API, model hot-swap, health checks and metrics, so your existing tools keep working and your team can operate it.

§03 measured proof

Numbers from my own runs.

fig. 01 Decode speed by context, tok/s
4K
  • tbq4_0 55.3
  • planar3_0 53.9
32K
  • tbq4_0 51.5
  • planar3_0 50.6
262K
  • tbq4_0 77
  • planar3_0 ~47
RTX 4090, Qwen3.6-27B, MTP and flash attention on. 262K row: peak-speed configuration.

62 → 82tok/s

decode after my production tune (+31%)

4.125bits

per KV value, TBQ4 cache

16×

fewer CUDA graph re-captures than upstream

≤300MiB

pinned RAM for checkpoints, instead of 4.7 GB

Source: my fork’s README and my commits in llama.cpp-turboq-mtp; Qwen3.6-27B, one RTX 4090.

§04 deliverables

What you get.

  • A measured baseline: throughput, latency and memory on your workload
  • The tuned serving stack, as configuration and infrastructure as code
  • A before/after report with the numbers and what caused the gap
  • A runbook your team can operate without me

The math behind this.

All articles
§05 answers

Questions I get asked.

How much faster can inference get?

It depends on what is limiting you. On my own stack, tuning decode and the KV cache moved peak decode from 62 to 82 tok/s, +31%, and cut CUDA graph re-captures 16×. A stack with an obvious bottleneck often moves more; a well-tuned one moves less. The teardown gives you a number before you commit to a build.

Does quantisation hurt quality?

It can, which is why it is measured. I quantise to the lowest precision that still passes your evals and report the delta. A 4-bit cache is not free accuracy-wise; it is a trade you should see, not guess.

vLLM, llama.cpp or TensorRT-LLM?

The one that fits your hardware and your workload. I run llama.cpp in production on a 4090 and I read the kernels; I pick the engine after the load test, not before.

Can you optimise a hosted API?

Only at the edges: prompt shape, caching, routing between models, context discipline. If you do not control the serving stack, the biggest wins are usually in what you send, not in the engine.

How long does it take?

A teardown is one to two weeks. A tuning build is scoped after that, usually two to four weeks depending on how much of the stack changes.

Serving bill growing faster than revenue?

Describe your problem

30-minute technical call with the engineer who builds it. No sales team, no slides.