Where we are in the series
Class 1 and 2 taught the model to imitate.
Both are imitation, and it has a ceiling, the demonstrator
SFT on traces
Distillation
GRPO
RL Envs
What changes today: the signal
SFT → a target token at every position (dense)
Distillation → the teacher’s distribution at every position (denser)
RL → one number per attempt: the reward (sparse)
The chess player from Class 2: stop studying the coach’s games and play your own. Nobody shows you the move, you get one signal per game (win/lose). That is enough to improve beyond anyone you could imitate.
The signal tells the model how well it did
How GRPO works
We will cover
The loop
Policy, rollout, reward, update: and what one GRPO round actually does
The group baseline
Where the learning signal comes from, and why there is no value model
Two guardrails
Clipping bounds one reused batch; the KL penalty bounds the whole run
The reward contract
What a verifier can see, and why it must produce variation
Running it
The same loop in TRL, and how to read a run that is going wrong
Reinforcement learning, in three functions
Policy
The model. Maps a prompt to a distribution over completions. This is what you train.
Rollout
Sampling completions from the policy. The model's own attempts at the task.
Reward
A function from a completion to a number. Higher is better.
Distillation Landscape
One GRPO training round
5
The group is the baseline
Scores become advantages
advantage = (reward - group mean) / (group std)
7
Two guardrails, two time scales
epsilon bounds one reused batch. beta bounds the whole run — and its default of 0.0 loads no reference model at all.
The whole update is a few lines
ONE STEP PER ROLLOUT
rewards = torch.tensor(
[reward(r) for r in responses]
)
advantages = (rewards - rewards.mean()) / (
rewards.std() + 1e-4
)
seq_logp = (token_logp * comp_mask).sum(dim=1)
loss = -(advantages * seq_logp).mean()
REUSING A ROLLOUT ADDS A CLIP
coef_1 = torch.exp(
per_token_logps - old_per_token_logps
)
coef_2 = torch.clamp(
coef_1, 1 - epsilon, 1 + epsilon
)
per_token_loss = -torch.min(
coef_1 * advantages, coef_2 * advantages
)
Positive advantage raises the completion's log-probability, negative lowers it, and the size sets how hard.
Generation is the slow part, so rollouts get reused. At a single step the ratio is one and the clip does nothing.
The reward is a testable contract
A fluent derivation with a wrong result fails. A lucky guess with the right result passes. The optimizer can only improve what the reward distinguishes.
| | Reward | ||
Answer | Reasoning | Format | Accuracy | Total |
<answer>72</answer> | Sound | 1.0 | 2.0 | 3.0 |
<answer>72</answer> | Faulty, a lucky guess | 1.0 | 2.0 | 3.0 |
<answer>71</answer> | Fluent, with one wrong step | 1.0 | 0.0 | 1.0 |
"The answer is 72" with no tags | Sound | 0.0 | 0.0 | 0.0 |
<answer>72</answer> followed by more text | Sound | 0.0 | 0.0 | 0.0 |
The reward must create variation
Too hard
nothing is ever solved
0
0
0
0
advantage = 0
The model cannot reach the reward yet, so every attempt sits on the mean.
Too easy
everything is solved
2
2
2
2
advantage = 0
The task is already solved, and there is nothing left for the reward to separate.
Usable
the group disagrees
0
1
1
2
signal
Some attempts beat the mean and some fall short, so the step has a direction to move in.
Fixed preferences or fresh samples
13
What GRPO drops: the critic
PPO: train a value model
GRPO: use the sampled group
The same loop wrapped in GRPOTrainer and GRPOConfig
trainer = GRPOTrainer(
model="Qwen/Qwen3-0.6B",
reward_funcs=[format_reward, accuracy_reward],
args=GRPOConfig(num_generations=8, beta=0.0),
train_dataset=dataset,
)
Reading the run
Experiments
GRPO in
The smallest GRPO example (exp 1)
Before any real run, let’s reward the length of the completion → be close to 20 characters long
GRPO optimizes exactly the number (reward) you return
Quickly, the model (Qwen3-0.6B) optimizes for that goal
GRPO for coding example (exp 2)
MBPP dataset: short Python problems, each with unit tests
The model writes the function. We run the asserts.
reward = fraction that pass
This is RLVR: reinforcement learning with verifiable rewards. No reward model, no human labels
GRPO for coding example (exp 2)
Does the task give signal? (exp 2)
If every attempt in a group gets the same reward, the advantage is 0 → nothing to learn. You want the task at the edge of the model’s ability
Before training Qwen3-0.6B on MBPP, k=4 attempts per prompt → mean reward 0.22, and 10 of 20 prompt groups mixed
We trained 3 different versions (num_generations)
Three different versions (exp 2)
The knob is the num_generations (group size)
k=2, k=8, k=16
k=2 → compares each attempt against a single sibling
k=16 → compares against a crowd
reward → the target-task signal. Should climb 📈
Failed example (exp 3)
The reward can be gamed
Reward hacking → model maximizes the reward ≠ solving task Reward 📈(training), real quality 📉(test)
Recap
Imitation to outcomes
SFT copies demonstrations. RL trains on results.
Group sampling
Sample a group of completions per prompt. Score each one.
Relative advantage
The group mean is the baseline. No reward model, no critic.
Verifiable rewards
Rewards are plain functions that check the output.
TRL on HF Jobs
GRPOTrainer runs the whole loop on Hugging Face Jobs.
Read the curves
Reward, KL, and completion length reveal hacking, collapse, and length gaming.