https://www.trefon.com/notes/Technical articles on LLM inference, model training and agent harnesses: KV cache, quantisation, attention, speculative decoding, RL, evals. Every number shown.en-GBhttps://www.trefon.com/notes/agent-context-management/https://www.trefon.com/notes/agent-context-management/Tue, 27 Oct 2026 08:00:00 GMTA 262K context isn't a licence to dump everything in. Each token of context costs speed on every step. How to budget context for an agent, truncate tool output without losing the signal, compact history without breaking the prefix cache, isolate work in sub-agents, and detect loops before they burn the budget.deep diveagentscontextharnesseshttps://www.trefon.com/notes/agent-eval-statistics/https://www.trefon.com/notes/agent-eval-statistics/Tue, 27 Oct 2026 08:00:00 GMTLeaderboards rank models by a few points; the noise is usually bigger. Confidence intervals on pass rates, worked on my own 49-scenario benchmark, how many scenarios you'd need to separate two models, why paired tests are worth the effort, and why pass^k beats pass@k for agents.deep diveevalsstatisticsagentsbenchmarkshttps://www.trefon.com/notes/flashattention-online-softmax/https://www.trefon.com/notes/flashattention-online-softmax/Tue, 27 Oct 2026 08:00:00 GMTAttention is three lines of math and one big memory problem. How the online softmax makes tiling exact, what FlashAttention never writes to memory, why decode attention is a stream over the KV cache, and why a quantised cache only pays off when dequant is fused into the kernel.deep diveattentionFlashAttentionCUDAKV cachehttps://www.trefon.com/notes/attention-is-all-you-need/https://www.trefon.com/notes/attention-is-all-you-need/Tue, 27 Oct 2026 08:00:00 GMTThe 2017 Google paper that introduced the transformer: the problem it set out to solve, what it actually proposed (scaled dot-product attention, multi-head attention, sinusoidal positions, an encoder-decoder for translation), the numbers it reported, what later models changed, and how a translation paper became the basis of modern AI.deep divetransformerattentionarchitecturehistoryhttps://www.trefon.com/notes/constrained-decoding-grammars-json/https://www.trefon.com/notes/constrained-decoding-grammars-json/Tue, 27 Oct 2026 08:00:00 GMTHow engines guarantee valid output by masking logits: finite-state machines for regular formats, pushdown automata for nested JSON, why tokens and grammar symbols don't line up, what it costs per step, and why constraining too early can make the model reason worse.deep diveconstrained decodingJSONtool callingllama.cpphttps://www.trefon.com/notes/continuous-batching-pagedattention/https://www.trefon.com/notes/continuous-batching-pagedattention/Tue, 27 Oct 2026 08:00:00 GMTWhy batching is nearly free for memory-bound decode, where it stops being free, why KV memory rather than compute caps the batch, and what continuous batching, PagedAttention and chunked prefill each fix. Worked on a 27B model on one RTX 4090.deep divebatchingPagedAttentionservinginferencehttps://www.trefon.com/notes/cuda-graphs-decode/https://www.trefon.com/notes/cuda-graphs-decode/Tue, 27 Oct 2026 08:00:00 GMTA decode step is hundreds of small kernels, and launching each one costs CPU time. How CUDA graphs capture and replay the step, why a growing KV cache keeps invalidating them, and what raising the KV pad floor to 4096 changed in my fork.deep diveCUDAinferencellama.cpphttps://www.trefon.com/notes/decode-memory-bound-speculative/https://www.trefon.com/notes/decode-memory-bound-speculative/Tue, 27 Oct 2026 08:00:00 GMTSingle-stream decode speed is set by memory bandwidth, not FLOPs. The bandwidth ceiling for a 17 GB model on a 4090, the math of speculative decoding and MTP, how draft depth trades off against acceptance, and why the draft sampler matters.deep diveinferencespeculative decodingMTPCUDAhttps://www.trefon.com/notes/decoder-only-transformer/https://www.trefon.com/notes/decoder-only-transformer/Tue, 27 Oct 2026 08:00:00 GMTPart 1 of the architecture series. The causal block that GPT, Llama, Qwen and Gemma are built from: masked self-attention, the MLP, residuals and norms, where the parameters live (12·d² per layer, checked against GPT-2), why one stack serves both prompt and generation, and why this design won.deep divetransformerarchitecturedecoder-onlyhttps://www.trefon.com/notes/deepseek-architecture-moment/https://www.trefon.com/notes/deepseek-architecture-moment/Tue, 27 Oct 2026 08:00:00 GMTHow a Chinese lab trained a frontier-level open model on export-restricted GPUs, and why one week in January 2025 wiped hundreds of billions off Nvidia. Multi-head latent attention and its 57× smaller cache, fine-grained MoE with shared experts, auxiliary-loss-free balancing, multi-token prediction, FP8 training, and R1's pure-RL reasoning.deep diveDeepSeekMoEarchitectureRLhttps://www.trefon.com/notes/dense-vs-moe/https://www.trefon.com/notes/dense-vs-moe/Tue, 27 Oct 2026 08:00:00 GMTThe architecture-level comparison: how a router picks experts, why load balancing is the hard part, capacity factors and dropped tokens, how much of each model actually runs per token (from Mixtral's 28% to DeepSeek-V3's 5.5%), what MoE buys in training compute, and what it costs in memory, communication and agent behaviour.deep diveMoEarchitecturetraininghttps://www.trefon.com/notes/diffusion-models-explained/https://www.trefon.com/notes/diffusion-models-explained/Tue, 27 Oct 2026 08:00:00 GMTHow diffusion models generate by learning to undo noise: the forward process in one formula, why predicting the noise is the training objective, how sampling went from 1,000 steps to a handful, classifier-free guidance, latent diffusion's 48× compression, transformer backbones and flow matching, and diffusion for text.deep divediffusiongenerative modelsarchitecturehttps://www.trefon.com/notes/distillation-kl-on-policy/https://www.trefon.com/notes/distillation-kl-on-policy/Tue, 27 Oct 2026 08:00:00 GMTHow a small model learns from a big one: soft targets and the T² factor, forward vs reverse KL and why the direction matters, sequence-level vs logit distillation, the storage cost of teacher logits, and why on-policy distillation fixes the student's own mistakes.deep divedistillationtrainingKL divergencehttps://www.trefon.com/notes/encoder-decoder-transformer/https://www.trefon.com/notes/encoder-decoder-transformer/Tue, 27 Oct 2026 08:00:00 GMTPart 3 of the architecture series. The original transformer layout: an encoder that reads the input both ways, a decoder that writes with cross-attention into it, T5's span corruption, where the extra parameters go, and why the design still wins for speech, translation and other tasks where the input dwarfs the output.deep divetransformerarchitectureT5Whisperhttps://www.trefon.com/notes/encoder-only-transformer/https://www.trefon.com/notes/encoder-only-transformer/Tue, 27 Oct 2026 08:00:00 GMTPart 2 of the architecture series. Bidirectional attention, the masked-language-model objective with its 80/10/10 rule, why encoders can't generate but are still the right tool for classification, embeddings and rerankers, and what modern encoders such as ModernBERT changed.deep divetransformerarchitectureBERTembeddingshttps://www.trefon.com/notes/hybrid-gated-deltanet-linear-attention/https://www.trefon.com/notes/hybrid-gated-deltanet-linear-attention/Tue, 27 Oct 2026 08:00:00 GMTLinear attention turns the KV cache into a fixed-size state. The delta rule fixes what it forgets badly, and gating adds decay. Why hybrids keep a few full-attention layers, and what a recurrent state changes for prefix reuse, speculative rollback and long sessions.deep diveGated DeltaNetlinear attentionarchitectureKV cachehttps://www.trefon.com/notes/kv-cache-math/https://www.trefon.com/notes/kv-cache-math/Tue, 27 Oct 2026 08:00:00 GMTThe weights are a fixed cost; the KV cache grows with every token and decides what fits. Per-token size from first principles, worked on a 27B model at 262K context, why keys are harder to quantise than weights, and why 3-bit is the risk axis.deep diveKV cachequantizationattentionllama.cpphttps://www.trefon.com/notes/lora-qlora-math/https://www.trefon.com/notes/lora-qlora-math/Tue, 27 Oct 2026 08:00:00 GMTLow-rank adaptation from the matrix up: how many parameters a rank-16 adapter really trains, what alpha does, why full fine-tuning a 7B model needs over 100 GB and QLoRA fits a 27B base in 14 GB, and what LoRA is good and bad at.deep diveLoRAQLoRAfine-tuningtraininghttps://www.trefon.com/notes/matmul-tensor-cores-accelerators/https://www.trefon.com/notes/matmul-tensor-cores-accelerators/Tue, 27 Oct 2026 08:00:00 GMTAlmost all of an LLM's arithmetic is matrix multiplication. What a matmul costs in FLOPs and bytes, the roofline model and the ridge point of real GPUs, why tiling exists, how tensor cores and systolic arrays execute matrix math in hardware, and why precision formats keep shrinking.deep diveGPUtensor coresmatmulhardwarehttps://www.trefon.com/notes/mcp-under-the-hood/https://www.trefon.com/notes/mcp-under-the-hood/Tue, 27 Oct 2026 08:00:00 GMTThe Model Context Protocol on the wire: the JSON-RPC 2.0 messages, the initialize handshake and capability negotiation, tools, resources and prompts, stdio vs Streamable HTTP, what every tool schema costs in context, and the security model you have to bring yourself.deep diveMCPagentsharnessesJSON-RPChttps://www.trefon.com/notes/moe-single-gpu/https://www.trefon.com/notes/moe-single-gpu/Tue, 27 Oct 2026 08:00:00 GMTA mixture-of-experts model costs memory by its total parameters and bandwidth by its active ones. What that means on one 24 GB card, how expert offload to system RAM changes the ceiling, why batching helps MoE less than dense models, and how MoE did in my agent benchmark.deep diveMoEinferencellama.cpphttps://www.trefon.com/notes/prefix-caching-kv-reuse/https://www.trefon.com/notes/prefix-caching-kv-reuse/Tue, 27 Oct 2026 08:00:00 GMTAgents resend the whole conversation on every turn. Prefix caching skips the part the server has already computed, and the fork's README records a long-context turn going from ~46 s to ~460 ms. How it works, what breaks it, and why hybrid models need checkpoints.deep diveprefix cachingKV cacheagentsinferencehttps://www.trefon.com/notes/reasoning-chain-of-thought/https://www.trefon.com/notes/reasoning-chain-of-thought/Tue, 27 Oct 2026 08:00:00 GMTFrom 'let's think step by step' to models trained to think: why writing intermediate steps adds computation a transformer can't do in one pass, how self-consistency and RL turned it into test-time compute, what an 8,192-token thinking budget costs in seconds, whether the visible reasoning is faithful, and what it changes for agents.deep divereasoningchain of thoughtRLagentshttps://www.trefon.com/notes/rag-retrieval-math/https://www.trefon.com/notes/rag-retrieval-math/Tue, 27 Oct 2026 08:00:00 GMTRetrieval decides what the model gets to read, and most RAG failures happen there. Cosine similarity and what normalisation buys, how much memory a million chunks costs at each precision, how HNSW trades recall for speed, why hybrid BM25 plus dense search wins, and where a reranker earns its cost.deep diveRAGembeddingsretrievalsearchhttps://www.trefon.com/notes/rl-ppo-dpo-grpo/https://www.trefon.com/notes/rl-ppo-dpo-grpo/Tue, 27 Oct 2026 08:00:00 GMTWhat reinforcement learning optimises after fine-tuning, written as the actual objectives: PPO's clipped surrogate with a KL leash, DPO's preference loss without a reward model, and GRPO's group-relative advantage without a critic. A worked GRPO example, and why verifiable rewards changed the game for code.deep diveRLGRPODPOtraininghttps://www.trefon.com/notes/rnn-lstm-to-linear-attention/https://www.trefon.com/notes/rnn-lstm-to-linear-attention/Tue, 27 Oct 2026 08:00:00 GMTPart 4 of the architecture series. The recurrent update, why gradients vanish or explode over long sequences (0.9¹⁰⁰ is 2.7 × 10⁻⁵), how LSTM gates fixed it, why transformers replaced RNNs anyway, and how state-space models and linear attention brought constant-memory recurrence back into today's hybrid LLMs.deep diveRNNLSTMarchitecturelinear attentionhttps://www.trefon.com/notes/rope-yarn-long-context/https://www.trefon.com/notes/rope-yarn-long-context/Tue, 27 Oct 2026 08:00:00 GMTHow rotary position embeddings encode position as rotation, what the wavelength of each dimension means for long context, and how position interpolation, NTK-aware scaling and YaRN stretch a model trained on 32K toward 262K.deep diveRoPElong contextattentionhttps://www.trefon.com/notes/sampling-temperature-top-p-min-p/https://www.trefon.com/notes/sampling-temperature-top-p-min-p/Tue, 27 Oct 2026 08:00:00 GMTWhat each sampler does to the probability distribution, worked on real numbers: temperature, top-k, top-p, min-p and repetition penalties. Why the order of the chain matters, why penalties break code and JSON, and how sampling settings show up in agent loops and decode speed.deep divesamplinginferenceagentshttps://www.trefon.com/notes/tokenizers-bpe-byte-level/https://www.trefon.com/notes/tokenizers-bpe-byte-level/Tue, 27 Oct 2026 08:00:00 GMTHow BPE builds a vocabulary from merges, why byte-level BPE never fails on unknown text, what vocabulary size costs in parameters and saves in sequence length, why Polish and code cost more tokens than English prose, and where byte-level models without a tokenizer stand.deep divetokenizerBPEbyte-levelhttps://www.trefon.com/notes/q4-k-m-quantization-explained/https://www.trefon.com/notes/q4-k-m-quantization-explained/Tue, 27 Oct 2026 08:00:00 GMTQ4_K_M byte by byte, from the ggml source: 256-weight super-blocks, 6-bit scales and mins, which tensors get upgraded to Q6_K and why, what an outlier costs, and where rotation comes in. Plus the size and speed of a 27B model at every k-quant.deep divequantizationGGUFllama.cpphttps://www.trefon.com/notes/training-cost-6nd-memory/https://www.trefon.com/notes/training-cost-6nd-memory/Tue, 27 Oct 2026 08:00:00 GMTBack-of-envelope training cost you can trust: the 6ND rule for compute, what Chinchilla says about tokens per parameter, why Adam costs 16 bytes per parameter, and where activation memory goes. Worked for a 125M model, a 1B model and a 4.5B fine-tune on one RTX 4090.deep divetrainingscaling lawsGPUhttps://www.trefon.com/notes/llama-cpp-turboq-262k-rtx-4090/https://www.trefon.com/notes/llama-cpp-turboq-262k-rtx-4090/Mon, 26 Oct 2026 08:00:00 GMTThe engine already kept a 4.125-bit KV cache inside flash attention. Running coding agents on it all day took async checkpoints, fewer CUDA graph re-captures, a stack of MTP fixes and a sampler tune worth +31%.inferencellama.cppCUDAMTPKV cachehttps://www.trefon.com/notes/amber-bench-harness-not-model/https://www.trefon.com/notes/amber-bench-harness-not-model/Mon, 26 Oct 2026 08:00:00 GMTA model leaderboard can't tell you whether your agent loop is broken. I built a benchmark that scores the harness and the model as a pair. The first thing it found was bugs in my own harness.harnessesevalsC++amber