RL for LLMs: PPO, DPO and GRPO, the objectives side by side

Supervised fine-tuning teaches a model to imitate examples. Reinforcement learning teaches it to prefer outcomes: answers that pass tests, that a reviewer would pick, that follow the format. Three objectives cover most of what’s used today. Here they are as the math, not the marketing.

The common frame

The model is a policy π_θ that produces an answer y to a prompt x. A reward r(x, y) scores the answer. We want answers with higher reward, but without drifting so far from the starting model, the reference π_ref, that the model forgets language or finds a cheap way to game the reward. So almost every method optimises:

maximise   E[ r(x, y) ]  −  β · KL( π_θ ‖ π_ref )

The methods differ in where r comes from and how the gradient is estimated.

PPO: the clipped surrogate

PPO (Schulman et al., 2017) is what classic RLHF used (Ouyang et al., 2022). A reward model is trained on human preference pairs with the Bradley–Terry loss, P(y_w ≻ y_l) = σ(r(y_w) − r(y_l)). Then the policy is optimised on that reward, minus a per-token KL penalty to the reference. The update is clipped so that one batch can’t move the policy too far:

ratio_t = π_θ(a_t | s_t) / π_old(a_t | s_t)
L = E[ min( ratio_t · A_t ,  clip(ratio_t, 1 − ε, 1 + ε) · A_t ) ]      ε ≈ 0.2

A_t, the advantage, says how much better a token was than expected. Estimating it needs a value model, a critic. That makes four models in memory: the policy, the reference, the reward model and the critic. It’s powerful, but it’s expensive and fiddly to tune.

DPO: no reward model, no sampling

DPO (Rafailov et al., 2023) shows that the KL-regularised objective has a closed-form optimal policy. That lets you skip the reward model entirely and train directly on preference pairs, a preferred answer y_w and a rejected one y_l:

L = −log σ( β · [ log π_θ(y_w|x)/π_ref(y_w|x)  −  log π_θ(y_l|x)/π_ref(y_l|x) ] )

It’s supervised-style training on a fixed dataset: no sampling during training, no critic, only the policy and a frozen reference. It’s cheap and stable. The limit is that it’s offline: it learns from the pairs you have, not from the model’s own new attempts, and it needs good preference data.

GRPO: group-relative advantage, no critic

GRPO (Shao et al., 2024, introduced with DeepSeekMath and used for DeepSeek-R1) keeps online sampling but drops the critic. For each prompt it samples a group of G answers, scores each one, and uses the group itself as the baseline:

A_i = ( r_i − mean(r_1..G) ) / std(r_1..G)

That advantage goes into the same clipped surrogate as PPO, with a KL term to the reference. No value network means one model fewer in memory and nothing to train alongside the policy.

A worked example. Eight answers to one coding prompt, scored 1 if the tests pass and 0 if they don’t:

rewards  = [1, 0, 0, 1, 1, 0, 0, 0]
mean     = 0.375    std = 0.484
A(pass)  = +1.29    A(fail) = −0.775

Passing answers are pushed up, failing ones down, relative to how the model does on this prompt today.

Notice the edge cases. If all eight pass, or all eight fail, the standard deviation is zero and the prompt teaches nothing. Prompts the model always solves or never solves are wasted compute, so the useful training set sits at the edge of the model’s ability. Later work, such as Dr. GRPO (Liu et al., 2025), also examines biases from the std normalisation and the length averaging. The basic form above is the place to start.

Verifiable rewards

A learned reward model can be gamed: the policy finds outputs the reward model likes that humans wouldn’t. For some domains you don’t need one, because the reward is a fact:

• Code: do the tests pass?
• Maths: does the final answer match?
• Format: does the tool call parse against the schema?

Training against rewards like these (RLVR, reinforcement learning with verifiable rewards) is what made the recent jumps in reasoning and coding models possible. It’s also the shape of the plan for my compass model: fine-tune a small Qwen for terminal and agent work, distil it, then run RL per language and stack, the kind of setup where tests can decide the reward.

Choosing

	Needs a reward model	Samples during training	Critic	Best for
PPO (RLHF)	yes	yes	yes	general preference tuning, when you can afford it
DPO	no	no	no	preference pairs you already have; cheap alignment
GRPO	no, if the reward is verifiable	yes	no	code, maths and tool use, where you can check the answer

Rules I use

1. Supervised fine-tuning first. RL sharpens behaviour the model can already produce sometimes. It doesn’t invent it.
2. Make the reward verifiable where you can. Tests beat judges.
3. Filter prompts to the frontier. Always-pass and always-fail prompts carry no GRPO signal.
4. Watch the KL and the length. Reward going up while answers balloon, or the KL exploding, usually means the reward is being gamed.

Sources: Schulman et al., Proximal Policy Optimization (2017). Ouyang et al., Training language models to follow instructions with human feedback (2022). Rafailov et al., Direct Preference Optimization (2023). Shao et al., DeepSeekMath (2024). DeepSeek-AI, DeepSeek-R1 (2025). Liu et al., Understanding R1-Zero-Like Training (2025).