Decoder-only transformers: the architecture behind every chat model

GPT, Llama, Qwen, Gemma, DeepSeek: almost every model you’d call an LLM today is a decoder-only transformer. It’s the simplest of the transformer family, a single stack trained to predict the next token. That simplicity is why it won. This is part 1 of four; the others cover encoder-only, encoder-decoder and recurrent models.

The block

The model is token embeddings, followed by L identical blocks, followed by a projection back to vocabulary logits. Each block does two things, each wrapped in a residual connection with a normalisation in front (pre-norm, the modern default):

x = x + Attention(Norm(x))      tokens look at earlier tokens
x = x + MLP(Norm(x))            each token is transformed on its own

The residual stream x is the model’s working memory. Every block reads from it and adds its update back.

Causal attention

“Decoder” means one thing: each position can only attend to itself and earlier positions. A triangular mask sets the attention scores for future positions to −∞ before the softmax. That’s what makes next-token training possible on whole sequences at once. Position 500 is predicted from tokens 0 to 499, and every position in a training sequence is a training example in the same pass.

It’s also what makes the KV cache possible at inference. Because earlier tokens never look forward, their keys and values never change when new tokens arrive, so they can be computed once and kept. The KV cache note is about exactly that store.

Where the parameters live

With hidden size d and a standard 4× MLP, one block holds about:

attention:  Q, K, V, O projections     4 · d²
MLP:        up d→4d, down 4d→d         8 · d²
per block                             12 · d²
model       ≈ 12 · L · d²  +  vocabulary embeddings

Checked against GPT-2 small (L = 12, d = 768, a vocabulary of 50,257 and 1,024 positions):

blocks       12 × 12 × 768²  =  84.9M
token embeddings  50,257 × 768  =  38.6M
positions     1,024 × 768  =   0.8M
total                          ≈ 124.3M     (published: 124M)

Modern variants move the numbers but not the shape. A SwiGLU MLP uses three matrices with a narrower hidden size, about 8/3 · d, which keeps the MLP near 8·d². Grouped-query attention shrinks the K and V projections. Llama-2-7B is 32 blocks at d = 4096 with an MLP width of 11,008, which comes to 6.74B with its untied embeddings. Roughly two-thirds of a dense model’s parameters are in the MLPs.

One stack for prompt and generation

The same blocks process your prompt and generate the answer. Prefill runs the whole prompt through in parallel, compute-bound, filling the KV cache. Decode then adds one token at a time, memory-bound, reading the cache (the decode note covers why). There’s no separate encoder, no cross-attention, and one cache to manage.

That uniformity is what makes the serving tricks work:

• Prefix caching: a prompt is just the start of a sequence, so shared starts can be reused.
• Speculative decoding: checking draft tokens is just a forward pass over a few more positions.
• Continuous batching: every request is the same shape of work.

Why it won

The decoder-only design wasn’t obviously the best at first. Encoder-decoder models were often stronger on supervised tasks per parameter. It won on scale and simplicity:

• Every token trains the model. Next-token prediction on raw text needs no labels and no masking scheme, and it uses every position of every document.
• One objective covers everything. Translation, summarisation, code and chat are all “continue this text” with a different prompt, so one model and one training run serve them all.
• It scales cleanly. One stack, one loss, predictable scaling laws, and the serving stack above.

Bidirectional context, the encoder’s main advantage, matters less once a model is big enough and the prompt itself carries the task.

Rules I use

1. Estimate parameters as 12·L·d² plus embeddings, then check against the config. Big mismatches usually mean a non-standard MLP, a different attention layout, or experts.
2. Remember most parameters are in the MLP. That’s where quantisation and LoRA spend their budget (the quantisation and LoRA notes have the details).
3. Treat prefill and decode as different workloads. Same weights, different bottleneck.

Sources: Vaswani et al., Attention Is All You Need (2017). Radford et al., GPT-2 (2019). Touvron et al., Llama 2 (2023). Shazeer, GLU Variants Improve Transformer (2020). Parameter counts are computed from the published configs.