What training an LLM costs: 6ND FLOPs, 16 bytes per parameter, Chinchilla

Before training anything, you can predict roughly what it will cost in hours and in memory, from three numbers: parameters, tokens and GPU throughput. Here’s the arithmetic I use, and where each number comes from.

Compute: 6ND

Training a dense transformer with N parameters on D tokens costs about:

C ≈ 6 · N · D   FLOPs

The forward pass is about 2N FLOPs per token, a multiply and an add per weight. The backward pass is about twice that, 4N, because it computes gradients for both activations and weights. Attention adds a term that grows with context length, but at typical training lengths, for models of this size, it’s a small correction (Kaplan et al., 2020).

To turn FLOPs into hours, you need the throughput you’ll actually get. Peak tensor throughput is never reached. Model FLOPs utilisation (MFU) of 30–50% is typical for a well-tuned single-GPU run. For an RTX 4090, assume about 165 TFLOPS peak in bf16 with fp32 accumulation, and 40% utilisation, so 66 TFLOPS sustained:

Run	N	D	FLOPs	Time at 66 TFLOPS
Small model, compute-optimal	125M	2.5B tokens	1.9 × 10¹⁸	~7.9 hours
1B model, compute-optimal	1B	20B tokens	1.2 × 10²⁰	~21 days
4.5B fine-tune	4.5B	1B tokens	2.7 × 10¹⁹	~4.7 days

The first row is the kind of model my llm-mini repo trains from scratch to show how the pieces fit. The last row is the shape of a fine-tuning run for a model like compass.

How many tokens: Chinchilla

Hoffmann et al. (2022) found that for a fixed compute budget, the loss-optimal model trains on about 20 tokens per parameter: 2.5B tokens for a 125M model, 20B for a 1B model. Earlier large models were heavily under-trained by this standard.

Two caveats matter in practice. Chinchilla optimises training compute, not deployment. If a model will serve billions of tokens, it’s worth training a smaller model far longer than 20 tokens per parameter, because inference is paid per use. Most recent open models do exactly that, at hundreds or thousands of tokens per parameter. And fine-tuning budgets are about the behaviour change you want, not about Chinchilla.

Memory: 16 bytes per parameter

Mixed-precision training with Adam stores, per parameter:

bf16 weights        2 bytes
bf16 gradients      2
fp32 master weights 4
Adam first moment   4
Adam second moment  4
                   ── 16 bytes per parameter   (ZeRO paper accounting)

For a 4.5B model, that’s 72 GB before a single activation, three times what one 24 GB card holds. The ways out:

• 8-bit optimiser states. Keeping Adam’s moments in 8 bits brings this to about 10 bytes per parameter, 45 GB here. Still too much.
• LoRA or QLoRA. Train adapters only, and the base model’s 16 bytes become 2, or about 0.5 in 4-bit. The LoRA note has the numbers.
• Offload or shard. Move optimiser states to CPU RAM (ZeRO-Offload), or shard across GPUs. That trades memory for bandwidth.

Activations: the part that scales with context

Activations depend on batch size b, sequence length s, hidden size h and attention heads a. Korthikanti et al. (2022) give the per-layer cost for a GPT-style block, without recomputation, as about s·b·h·(34 + 5·a·s/h) bytes.

Take a 4.5B-class configuration (h = 3072, a = 24) at s = 4096, b = 1. That’s about 2.4 GB per layer, almost all of it in the 5·a·s/h attention-score term. FlashAttention never materialises the attention scores, so that term essentially disappears and about 0.43 GB per layer remains. Gradient checkpointing cuts most of the rest by recomputing activations in the backward pass. It costs roughly a third more compute, but that’s usually the trade that makes a run fit.

Rules I use

1. Estimate before you start: 6ND at a realistic 30–50% utilisation, and 16 bytes per trainable parameter.
2. Train small models long if they’re going to serve. Chinchilla is about training cost, not total cost.
3. Fit memory with adapters first, then 8-bit optimisers, then checkpointing, then offload, in that order of pain.
4. Measure utilisation early. If a short run shows 15% MFU, fix the data loader or the kernels before committing to days of training.

Sources: Kaplan et al., Scaling Laws for Neural Language Models (2020). Hoffmann et al., Training Compute-Optimal Large Language Models (2022). Rajbhandari et al., ZeRO (2020). Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models (2022). Times are estimates at 66 TFLOPS sustained, not measurements.