Trefon.
ls ~/work Work

Built, measured, handed over.

Open-source engines you can read line by line, and client builds I can describe but not name. Every number here comes from a benchmark or a commit, and every repo is public.

§01 inference

A 27B model, 262K context, one RTX 4090.

I run and maintain a fork of llama.cpp-turboq, the engine that keeps a 4.125-bit KV cache inside the flash-attention kernel. That kernel, RotorQuant and the first MTP implementation are upstream work by Indras-Mirror and contributors. With an FP16 cache, 262K tokens of context alone would take two thirds of a 24 GB card; with TBQ4 it fits next to a 27B model. My part is making that hold up under real agent workloads, every day.

What I added

  1. 01 Async checkpoint capture through a pinned two-slot staging ring: the 32 prompt checkpoints of a 262K prefill no longer stall the compute stream, and pinned RAM stays under ~300 MiB instead of 4.7 GB.
  2. 02 A KV pad floor: 16× fewer CUDA graph re-captures than upstream’s hard-coded 256, plus an adaptive guard that throttles MTP draft depth when acceptance drops below 35%.
  3. 03 MTP and Gated DeltaNet fixes for long agent sessions: shadow-context sync, a stochastic draft sampler, rejection rollback for M-RoPE models, a breaker for tool-call loops and a guard against recurrent-state drift at high context.
  4. 04 A fused chunk-pipeline kernel for DeltaNet prefill.
  5. 05 In-process model hot-swap without a separate router, and OpenAI API extensions (reasoning effort, message-aware prompt trimming) so agent harnesses run against it unmodified.
  6. 06 A Metal port of the TBQ3 and TBQ4 KV cache with rotated-domain dequant, so the same cache runs on Apple silicon.

Built on: Indras-Mirror/llama.cpp-turboq-mtp ggml-org/llama.cpp

github.com/jtrefon/llama.cpp-turboq-mtp code ↗ → Read the lab note
fig. 01 KV cache at 262K context
  • FP16 16 GiB
  • tbq4_0 4.1 GiB
  • planar3_0 3.1 GiB
32,768 cached values per token: 2 bytes each at FP16, 66 bytes per 128 values with TBQ4.
fig. 02 Decode speed by context, tok/s
4K
  • tbq4_0 55.3
  • planar3_0 53.9
32K
  • tbq4_0 51.5
  • planar3_0 50.6
262K
  • tbq4_0 77
  • planar3_0 ~47
RTX 4090, Qwen3.6-27B, MTP and flash attention on. 262K row: peak-speed configuration.
decode after my production tune (+31%)
62 → 82tok/s
decode after my production tune (+31%)
fewer CUDA graph re-captures than upstream
16×
fewer CUDA graph re-captures than upstream
pinned RAM for checkpoints, instead of 4.7 GB
≤300MiB
pinned RAM for checkpoints, instead of 4.7 GB
context next to a 27B model on 24 GB
262Ktokens
context next to a 27B model on 24 GB

Source: the repo README and my commits in the fork

§02 harnesses

A harness you can measure.

amber is a native C++17 agent runtime: one plugin architecture for providers, tools and UI, multi-provider, MCP, sub-agents, a BitchX-style terminal UI and a headless CLI for Linux and macOS. Next to it I built amber-bench, so every architecture change shows up as a number instead of a feeling.

  • 49 scenarios in 10 suites: coding, terminal work, tool selection, prompt adherence, refactoring, review, delegation, agent failures and loop discipline.
  • Each scenario is scored 0–100 on correctness, efficiency, robustness and prompt adherence, weighted by difficulty and scaled to 1000.
  • The +52.5 jump between v2 and v3 came from three harness defects the benchmark itself surfaced: path normalisation in the oracle, bare-JSON extraction and an SSE-sink lifetime bug.
  • A hermetic mode with a fake LLM runs the suite deterministically in CI, with no model and no GPU.
amber, native C++ AI agent runtime: git clone github.com/jtrefon/amber; cd amber; make; ./amber
github.com/jtrefon/amber code ↗ → Live leaderboard: ten local models Full benchmark tables and method ↗ → Read the lab note
fig. 03 amber-bench, harness versions · Qwopus3.6-27B
  • v2 769.1 · 39/43
  • v3 821.6 · 41/43
Same model, same scenarios. Only the harness changed. Axis 700–1000.
fig. 04 Ten local models, one harness · score /1000
  • Qwen3.6-27B dense 816.2
  • Qwopus3.6-27B-v2-MTP 808.0
  • Gemma-4-31B 806.2
  • Gemma-4-12B Q4 780.0
  • Ornith-1.0-35B 770.0
  • Qwen3.6-27B MTP 768.8
  • Gemma-4-12B Q8 756.8
  • Qwen3.6-35B-A3B MoE 742.2
  • Qwen3.6-27B pi-reason. 723.3
  • Ornith-1.0-9B 716.6
49 scenarios each, serial, single GPU slot. Axis 700–1000.
§03 desktop

A native macOS IDE. No Electron.

compass IDE: file tree, code editor with a TypeScript service, terminal, and the agent chat reading project files
fig. 05 — compass, agent chat reviewing a TypeScript project

compass is an AI IDE written in Swift. Agentic coding runs on your own provider and keys; RAG over your codebase, fill-in-the-middle completion and chat run on-device and keep working offline. MIT licensed, installed with Homebrew.

  • Swift
  • on-device RAG
  • FIM completion
  • offline chat
  • your own keys
  • MIT
github.com/jtrefon/compass code ↗ jtrefon.github.io/compass ↗
§04 open source

More code you can read.

§05 client builds

Client work, described, not named.

Clients own what I build for them, so their names stay off this page. I can walk you through the architecture on the call.

  1. Harness handed over

    Off-plan real estate, Dubai

    Agentic sales platform: .NET core and API, Angular back office, React chat agent, multiple LLM providers routed through OpenRouter.

    • .NET
    • Angular
    • React
    • OpenRouter
  2. Harness handed over

    Inbound sales agents, several markets

    Agents written in PHP and deployed inside the client’s own application, wired to local scripts, SOPs and domain knowledge, running on the Gemini API. Fast, cheap and reliable.

    • PHP
    • Gemini API
  3. Models ongoing · NDA

    Document extraction, regulated accounting client

    Vision-model extraction pipeline. Ongoing, under NDA.

    • vision model
    • Python
  4. Models training

    compass model (in progress, open source)

    A 4.5B agentic coding model: Qwen fine-tuned for terminal and agent work, then distilled and RL-tuned per language and stack, trained on one RTX 4090, to ship inside compass.

    • Qwen
    • RL
    • distillation
    • RTX 4090

Have a problem like this?

Describe your problem

30-minute technical call with the engineer who builds it. No sales team, no slides.