Inference that runs where your data lives.
I build and tune the serving stack: model selection, quantisation, KV-cache strategy, multi-token prediction, OpenAI-compatible APIs so your existing tools keep working, and hardware sizing. On-prem, in your cloud account, or air-gapped.
What you get
- 01 A running inference service
- 02 Benchmarks on your workload: throughput, latency, memory
- 03 Infrastructure as code
- 04 A runbook your team owns
Proof
- C++ code ↗
llama.cpp-turboq-mtp
My fork of llama.cpp-turboq. I added async checkpoint capture, a KV pad floor, MTP and Gated DeltaNet stability fixes, hot model loading without a separate router, OpenAI API extensions for agent compatibility and a Metal port of the TBQ KV cache. Up to 82 tok/s with a 4.125-bit KV cache at 262K context on one RTX 4090.
github.com/jtrefon/llama.cpp-turboq-mtp
- Swift code ↗
compass
Local inference, on-device RAG and code completion working fully offline.
github.com/jtrefon/compass
Lab notes on this
- Private LLM A private LLM on your own hardware.
- 2026-10-27 Attention from the kernel's side: FlashAttention, online softmax and fused dequant
- 2026-10-27 Continuous batching and PagedAttention: serving many users from one GPU
- 2026-10-27 CUDA graphs in LLM decode: launch overhead, and why shape changes cost
- 2026-10-27 Decode is memory-bound: the ~59 tok/s ceiling on an RTX 4090, and how speculative decoding gets past it
- 2026-10-27 Hybrid models: what Gated DeltaNet replaces in the KV cache
- 2026-10-27 KV cache math: what long context costs, and what a 4-bit cache buys
Scope check
Don’t hire me for this if
Your volume is low and nothing stops your data going to an API. The API is cheaper.
Have a problem like this?
Describe your problem30-minute technical call with the engineer who builds it. No sales team, no slides.