DeepSeek: the architecture behind V3 and R1, and the DeepSeek moment

On 27 January 2025, Nvidia lost about 17% of its value in a single trading day, roughly $590 billion, the largest one-day loss of market value for any company in US history. The trigger was an open-weights model from a Chinese lab: DeepSeek-R1, released a week earlier, matched OpenAI’s o1 on reasoning benchmarks. Its base model was reported to have cost about $5.6 million in compute to train. Here’s what DeepSeek actually built, and what the number did and didn’t mean.

Who DeepSeek is

DeepSeek is a Hangzhou lab spun out of High-Flyer, a quantitative hedge fund founded by Liang Wenfeng. It published unusually detailed technical reports and released its weights openly. Its earlier DeepSeek-V2 (May 2024) had already set off a price war among Chinese model APIs, by introducing the two architectural ideas that V3 scaled up.

Multi-head latent attention: shrink the cache

Standard multi-head attention caches a full key and value vector per head, per token, per layer. DeepSeek-V3 has 128 heads of 128 dimensions, so standard attention would cache 2 × 128 × 128 = 32,768 values per token per layer.

Multi-head latent attention (MLA) compresses each token’s keys and values into one shared latent vector of 512 dimensions, plus a small 64-dimensional key that carries the rotary position information. Only those are cached; full keys and values are reconstructed from the latent through learned up-projections:

standard MHA:  2 × 128 heads × 128 dims = 32,768 values per token per layer
MLA:           512 (latent) + 64 (RoPE key) =    576 values per token per layer   (≈ 57× smaller)

Across V3’s 61 layers at BF16, that’s about 69 KB of cache per token instead of about 3.8 MB. Long contexts and large batches that would be impossible with standard attention fit comfortably. Compare the techniques in the KV cache note: grouped-query attention shares KV heads, quantisation shrinks each value, and MLA attacks the problem in the architecture itself.

DeepSeekMoE: many small experts, plus shared ones

DeepSeek-V3 has 671B parameters in total, of which about 37B are active per token, 5.5%. Its MoE design differs from the classic one in two ways (the dense vs MoE note covers the basics):

• Fine-grained experts. Instead of a few large experts, each MoE layer has 256 small routed experts, and every token uses 8 of them. More, smaller experts allow more specific combinations and more specialised experts.
• A shared expert. Every token also passes through one always-on shared expert, which captures common knowledge so the routed experts don’t all have to relearn it.

Load balancing without an auxiliary loss. MoE routers tend to collapse onto a few favourite experts, so most models add an auxiliary loss that penalises imbalance, which also pulls against the main objective. V3 instead adds a per-expert bias to the routing scores and nudges it after each step: down for overloaded experts, up for idle ones. The bias affects only which experts are chosen, not how their outputs are weighted.

Multi-token prediction

V3 also trains with a multi-token prediction (MTP) objective: an extra sequential module predicts one more token beyond the next, which densifies the training signal. At inference, the MTP module can serve as a speculative drafter. The idea spread quickly. The Qwen models on my inference box ship MTP heads, and my fork’s speculative decoding runs on them (the decode note has the math).

FP8 training and the $5.6M number

V3 was trained on 14.8 trillion tokens, mostly in FP8 mixed precision, with fine-grained scaling to keep 8-bit values accurate. It used custom pipeline parallelism (DualPipe) that overlaps computation with the all-to-all communication that MoE requires, on 2,048 Nvidia H800s. The H800 is the export-compliant variant of the H100, with reduced chip-to-chip bandwidth. Squeezing communication was exactly where DeepSeek’s engineering went.

The report states the final training run at 2.788 million H800 GPU-hours, which comes to about $5.576 million at an assumed $2 per GPU-hour. It also says, explicitly, that this excludes the cost of prior research, ablations and experiments, and of course the cluster itself. The headline compared one training run’s rental cost with competitors’ total budgets. The real achievement wasn’t “AI for $5.6M”. It was frontier-level results at a fraction of the expected compute, through engineering, on restricted hardware.

R1: reasoning from reinforcement learning

DeepSeek-R1 (January 2025) took V3 and taught it to reason, in two steps that the report describes openly:

• R1-Zero: pure reinforcement learning on the base model with GRPO, no supervised examples first. The rewards were rule-based: the answer is correct, and the format is right. Long chains of thought, self-checking and the much-quoted “aha moment” of the model re-evaluating its own approach emerged from the reward alone (the RL note explains GRPO). R1-Zero’s outputs were hard to read and mixed languages.
• R1: a multi-stage pipeline. A small cold-start set of curated reasoning examples, then reasoning-focused RL, then supervised fine-tuning on samples filtered from the RL model plus general data, then a final RL stage for helpfulness and safety.

It was released under the MIT licence, together with distilled versions in which Qwen and Llama models from 1.5B to 70B were fine-tuned on about 800K samples generated by R1. The distillation note covers what that showed.

Why it was a “moment”

• Open weights at the frontier. For the first time, a model competitive with the best closed reasoning model could be downloaded, inspected and run by anyone.
• Efficiency as a strategy. It challenged the assumption that frontier AI requires the largest clusters, and it did so under export controls designed to slow exactly this.
• Method transparency. GRPO, rule-based rewards, MLA, MTP and the distillation recipe were all documented, and the open ecosystem adopted them within months.
• Price pressure. API prices for reasoning models fell sharply, and “good enough and cheap” became a real option.

What I take from it

1. Architecture still matters. MLA and fine-grained MoE are design choices that move memory and cost by orders of magnitude, not tuning.
2. Read the fine print on cost claims. A training run’s GPU-hours are not a lab’s budget.
3. Verifiable rewards are powerful. R1-Zero showed reasoning behaviour emerging from correctness rewards alone.

Sources: DeepSeek-AI, DeepSeek-V2 (2024), DeepSeek-V3 Technical Report (2024) and DeepSeek-R1 (2025). Dai et al., DeepSeekMoE (2024). Market figures are from contemporary reporting on 27 January 2025. Cache sizes are computed from the published V3 configuration.