Trefon.
ls ~/articles n=33

Engineering articles.

Inference, models and agent harnesses, from first principles and from the commit log. Every number is derived, measured or sourced, and the code is linked.

§01

Inference

n=11

  1. Attention from the kernel's side: FlashAttention, online softmax and fused dequant

    Attention is three lines of math and one big memory problem. How the online softmax makes tiling exact, what FlashAttention never writes to memory, why decode attention is a stream over the KV cache, and why a quantised cache only pays off when dequant is fused into the kernel.

  2. Continuous batching and PagedAttention: serving many users from one GPU

    Why batching is nearly free for memory-bound decode, where it stops being free, why KV memory rather than compute caps the batch, and what continuous batching, PagedAttention and chunked prefill each fix. Worked on a 27B model on one RTX 4090.

  3. CUDA graphs in LLM decode: launch overhead, and why shape changes cost

    A decode step is hundreds of small kernels, and launching each one costs CPU time. How CUDA graphs capture and replay the step, why a growing KV cache keeps invalidating them, and what raising the KV pad floor to 4096 changed in my fork.

  4. Decode is memory-bound: the ~59 tok/s ceiling on an RTX 4090, and how speculative decoding gets past it

    Single-stream decode speed is set by memory bandwidth, not FLOPs. The bandwidth ceiling for a 17 GB model on a 4090, the math of speculative decoding and MTP, how draft depth trades off against acceptance, and why the draft sampler matters.

  5. Hybrid models: what Gated DeltaNet replaces in the KV cache

    Linear attention turns the KV cache into a fixed-size state. The delta rule fixes what it forgets badly, and gating adds decay. Why hybrids keep a few full-attention layers, and what a recurrent state changes for prefix reuse, speculative rollback and long sessions.

  6. KV cache math: what long context costs, and what a 4-bit cache buys

    The weights are a fixed cost; the KV cache grows with every token and decides what fits. Per-token size from first principles, worked on a 27B model at 262K context, why keys are harder to quantise than weights, and why 3-bit is the risk axis.

  7. MoE on a single GPU: total vs active parameters, offload and bandwidth

    A mixture-of-experts model costs memory by its total parameters and bandwidth by its active ones. What that means on one 24 GB card, how expert offload to system RAM changes the ceiling, why batching helps MoE less than dense models, and how MoE did in my agent benchmark.

  8. Prefix caching: reusing the KV cache across turns and requests

    Agents resend the whole conversation on every turn. Prefix caching skips the part the server has already computed, and the fork's README records a long-context turn going from ~46 s to ~460 ms. How it works, what breaks it, and why hybrid models need checkpoints.

  9. RoPE from first principles, and how YaRN stretches a model's context

    How rotary position embeddings encode position as rotation, what the wavelength of each dimension means for long context, and how position interpolation, NTK-aware scaling and YaRN stretch a model trained on 32K toward 262K.

  10. Sampling math: temperature, top-p, min-p, and why agents loop

    What each sampler does to the probability distribution, worked on real numbers: temperature, top-k, top-p, min-p and repetition penalties. Why the order of the chain matters, why penalties break code and JSON, and how sampling settings show up in agent loops and decode speed.

  11. What Q4_K_M actually stores: weight quantisation from first principles

    Q4_K_M byte by byte, from the ggml source: 256-weight super-blocks, 6-bit scales and mins, which tensors get upgraded to Q6_K and why, what an outlier costs, and where rotation comes in. Plus the size and speed of a 27B model at every k-quant.
§02

Models

n=6

  1. Distillation: logit KL, temperature, and on-policy vs off-policy

    How a small model learns from a big one: soft targets and the T² factor, forward vs reverse KL and why the direction matters, sequence-level vs logit distillation, the storage cost of teacher logits, and why on-policy distillation fixes the student's own mistakes.

  2. LoRA and QLoRA math: rank, alpha, memory, and what actually trains

    Low-rank adaptation from the matrix up: how many parameters a rank-16 adapter really trains, what alpha does, why full fine-tuning a 7B model needs over 100 GB and QLoRA fits a 27B base in 14 GB, and what LoRA is good and bad at.

  3. Reasoning and chain of thought: why thinking in tokens works, and what it costs

    From 'let's think step by step' to models trained to think: why writing intermediate steps adds computation a transformer can't do in one pass, how self-consistency and RL turned it into test-time compute, what an 8,192-token thinking budget costs in seconds, whether the visible reasoning is faithful, and what it changes for agents.

  4. RL for LLMs: PPO, DPO and GRPO, the objectives side by side

    What reinforcement learning optimises after fine-tuning, written as the actual objectives: PPO's clipped surrogate with a KL leash, DPO's preference loss without a reward model, and GRPO's group-relative advantage without a critic. A worked GRPO example, and why verifiable rewards changed the game for code.

  5. Tokenizers: BPE, byte-level models, and what a token costs

    How BPE builds a vocabulary from merges, why byte-level BPE never fails on unknown text, what vocabulary size costs in parameters and saves in sequence length, why Polish and code cost more tokens than English prose, and where byte-level models without a tokenizer stand.

  6. What training an LLM costs: 6ND FLOPs, 16 bytes per parameter, Chinchilla

    Back-of-envelope training cost you can trust: the 6ND rule for compute, what Chinchilla says about tokens per parameter, why Adam costs 16 bytes per parameter, and where activation memory goes. Worked for a 125M model, a 1B model and a 4.5B fine-tune on one RTX 4090.
§03

Harnesses

n=6

  1. Agent context management: budgets, truncation, compaction, and knowing when to stop

    A 262K context isn't a licence to dump everything in. Each token of context costs speed on every step. How to budget context for an agent, truncate tool output without losing the signal, compact history without breaking the prefix cache, isolate work in sub-agents, and detect loops before they burn the budget.

  2. Agent eval statistics: is 816 vs 808 a real difference?

    Leaderboards rank models by a few points; the noise is usually bigger. Confidence intervals on pass rates, worked on my own 49-scenario benchmark, how many scenarios you'd need to separate two models, why paired tests are worth the effort, and why pass^k beats pass@k for agents.

  3. Constrained decoding: grammars and JSON schema so tool calls always parse

    How engines guarantee valid output by masking logits: finite-state machines for regular formats, pushdown automata for nested JSON, why tokens and grammar symbols don't line up, what it costs per step, and why constraining too early can make the model reason worse.

  4. MCP under the hood: JSON-RPC, transports, and capability negotiation

    The Model Context Protocol on the wire: the JSON-RPC 2.0 messages, the initialize handshake and capability negotiation, tools, resources and prompts, stdio vs Streamable HTTP, what every tool schema costs in context, and the security model you have to bring yourself.

  5. Retrieval math for RAG: embeddings, chunking, hybrid search and reranking

    Retrieval decides what the model gets to read, and most RAG failures happen there. Cosine similarity and what normalisation buys, how much memory a million chunks costs at each precision, how HNSW trades recall for speed, why hybrid BM25 plus dense search wins, and where a reranker earns its cost.

  6. amber-bench: benchmarking the harness, not the model

    A model leaderboard can't tell you whether your agent loop is broken. I built a benchmark that scores the harness and the model as a pair. The first thing it found was bugs in my own harness.
§04

Architectures

n=9

  1. Attention Is All You Need: the paper, the idea, and what it changed

    The 2017 Google paper that introduced the transformer: the problem it set out to solve, what it actually proposed (scaled dot-product attention, multi-head attention, sinusoidal positions, an encoder-decoder for translation), the numbers it reported, what later models changed, and how a translation paper became the basis of modern AI.

  2. Decoder-only transformers: the architecture behind every chat model

    Part 1 of the architecture series. The causal block that GPT, Llama, Qwen and Gemma are built from: masked self-attention, the MLP, residuals and norms, where the parameters live (12·d² per layer, checked against GPT-2), why one stack serves both prompt and generation, and why this design won.

  3. DeepSeek: the architecture behind V3 and R1, and the DeepSeek moment

    How a Chinese lab trained a frontier-level open model on export-restricted GPUs, and why one week in January 2025 wiped hundreds of billions off Nvidia. Multi-head latent attention and its 57× smaller cache, fine-grained MoE with shared experts, auxiliary-loss-free balancing, multi-token prediction, FP8 training, and R1's pure-RL reasoning.

  4. Dense vs mixture-of-experts: what you trade, from routing to serving

    The architecture-level comparison: how a router picks experts, why load balancing is the hard part, capacity factors and dropped tokens, how much of each model actually runs per token (from Mixtral's 28% to DeepSeek-V3's 5.5%), what MoE buys in training compute, and what it costs in memory, communication and agent behaviour.

  5. Diffusion models: from noise to images, and now to text

    How diffusion models generate by learning to undo noise: the forward process in one formula, why predicting the noise is the training objective, how sampling went from 1,000 steps to a handful, classifier-free guidance, latent diffusion's 48× compression, transformer backbones and flow matching, and diffusion for text.

  6. Encoder-decoder transformers: cross-attention, T5, Whisper, and when they still win

    Part 3 of the architecture series. The original transformer layout: an encoder that reads the input both ways, a decoder that writes with cross-attention into it, T5's span corruption, where the extra parameters go, and why the design still wins for speech, translation and other tasks where the input dwarfs the output.

  7. Encoder-only transformers: BERT, embeddings and rerankers

    Part 2 of the architecture series. Bidirectional attention, the masked-language-model objective with its 80/10/10 rule, why encoders can't generate but are still the right tool for classification, embeddings and rerankers, and what modern encoders such as ModernBERT changed.

  8. Matrix multiplication and how tensor accelerators work

    Almost all of an LLM's arithmetic is matrix multiplication. What a matmul costs in FLOPs and bytes, the roofline model and the ridge point of real GPUs, why tiling exists, how tensor cores and systolic arrays execute matrix math in hardware, and why precision formats keep shrinking.

  9. RNNs, from LSTM to linear attention: why recurrence lost and how it came back

    Part 4 of the architecture series. The recurrent update, why gradients vanish or explode over long sequences (0.9¹⁰⁰ is 2.7 × 10⁻⁵), how LSTM gates fixed it, why transformers replaced RNNs anyway, and how state-space models and linear attention brought constant-memory recurrence back into today's hybrid LLMs.
§05

Builds

n=1

  1. 262K context on one RTX 4090: what I added to llama.cpp-turboq

    The engine already kept a 4.125-bit KV cache inside flash attention. Running coding agents on it all day took async checkpoints, fewer CUDA graph re-captures, a stack of MTP fixes and a sampler tune worth +31%.

Older, non-technical writing is in the archive. Archive

Have a problem like this?

Describe your problem

30-minute technical call with the engineer who builds it. No sales team, no slides.