GRPO and Reinforcement Learning from Verifiable Rewards

The Problem

Lesson 09 built the RLHF pipeline: pretrain, then optimize a policy against a learned reward model with PPO. That pipeline has two expensive components.

The reward model is a liability. It is a neural network trained to imitate human preferences, and the policy is optimized to maximize its score. Any gap between "scores well" and "is actually correct" is an attack surface. Scale up training and the policy finds the gaps: verbose answers, confident hedging, sycophancy. Red-teaming the reward model never ends because the policy never stops probing it.

The critic is dead weight. PPO learns a value function V_\psi(s) just to compute advantages for the policy gradient. For LLM workloads that is a second full-size model sitting in GPU memory, receiving gradients, and — worse — its value estimates are nearly useless for sparse binary rewards ("the answer is right or it is not"). When most rollouts score 0, V_\psi(s) mostly learns to predict 0.

Yet for a huge class of prompts, none of this machinery is needed. A math problem has a ground-truth answer. A coding task has unit tests. A format requirement is a regular expression. If the reward can be computed rather than predicted, the reward model disappears, and if the reward is a single terminal scalar, the value network is the wrong tool anyway.

That is the setup for this lesson: Reinforcement Learning from Verifiable Rewards (RLVR) — rewards emitted by a program — optimized with Group Relative Policy Optimization (GRPO) — a PPO variant that replaces the critic with the empirical statistics of a group of samples drawn from the same prompt.

The Concept

RLVR: the reward is a program

A verifier is a function from a response to a reward:

where x is the prompt and y the response. Real systems stack several checkers: an accuracy reward (exact match against the ground truth, or unit tests passing), a format reward (the answer appears inside a designated tag, the CoT is separated from the final answer), and optionally soft terms (compilation success for code, runtime for kernels).

The verifier is exact, cheap to run at millions of rollouts, and — because it is code — it cannot be gradually drift-optimized the way a neural reward model can. It can still be hacked in the design space: if check only compares the final number, the policy may learn to skip the reasoning and guess. R1's answer to that is to require the reasoning trace in a fixed format and to train long enough that guessing has worse odds than reasoning. Verifier design is an engineering decision you will revisit per domain.

GRPO: the group is the baseline

PPO's advantage estimate A_t = Q(s_t, a_t) - V_\psi(s_t) needs the critic. GRPO (Shao et al. 2024, introduced with DeepSeekMath) instead samples a group of G responses \{y_1, \dots, y_G\} from the same prompt with the current policy, scores each with the verifier, and normalizes within the group:

The group mean is a Monte-Carlo estimate of the value of the state "this prompt" — exactly what the critic was approximating — computed for free from rollouts you needed anyway. When all G responses get the same reward (all pass or all fail), every advantage is 0 and the prompt contributes no gradient. That is correct behavior: a prompt that is trivially solved or hopelessly unsolved teaches nothing, and GRPO auto-focuses training on prompts in the band of usefulness.

The optimization step is PPO's clipped surrogate with the reference-policy KL added, using the k_3 estimator (unbiased, always non-negative):

Read the pieces as an engineer:

• \rho_{i,t} is the per-token importance ratio between the sampling policy and the policy being updated. Multiple epochs over the same group stay approximately on-policy thanks to the clip.
• The clip \epsilon \approx 0.2 bounds how far any single token can push the policy per update.
• \beta \approx 0.001–0.04 anchors the policy to the reference (the SFT checkpoint). Too large and the model refuses to move off its initial distribution; too small and the policy drifts into degenerate text that still satisfies the verifier.

The R1 recipe

The DeepSeek-R1 report (2025) is the reference pipeline for RLVR at scale:

1. Cold-start SFT — a few thousand long chain-of-thought examples so the base model produces parseable reasoning before RL starts. Pure RL from a base model produces language-mixing soup; the format must exist before it can be rewarded.
2. Reasoning-oriented RL — GRPO with accuracy + format rewards on math/code prompts. This is where the "aha" behaviors (self-verification, reflection, longer CoT on hard problems) emerge.
3. Rejection sampling + SFT — sample many, keep what the verifier accepts, fine-tune on the winners; then re-mix with non-reasoning data (writing, QA) so general ability does not rot.
4. RL for all scenarios — a second GRPO pass with helpfulness/harmlessness rewards on top of verifiable ones.

Kimi k1.5 (2025) contributes the length-control reward. Reasoning RL has a known pathology: the model discovers that thinking longer raises the odds of stumbling into the answer, so response length inflates and inference cost explodes. k1.5 reshapes the reward:

Correct answers earn more when they are shorter; wrong answers are penalized more when they are longer. Same RLVR machinery, smarter reward shape.

Build It

The lesson ships a complete GRPO trainer in NumPy: a tabular token-level policy, a real verifier, group sampling, clipped multi-epoch updates, the k_3 KL penalty, and the k1.5 length term.

The toy domain. Prompts supply three numbers (a permutation of 2, 3, 5) and a target. The policy emits a 5-token expression — two +/* operators interleaved with the three numbers, order free. The verifier evaluates with real operator precedence and pays out: accuracy reward for hitting the target, format reward for a grammatical expression, and the k1.5 length term (length is fixed here, so it acts as a small constant — but the function is implemented and tested, because plugging in variable-length decoding later must not change the trainer).

Some targets are hit by only a few of the 24 possible expressions (17 = 2+3*5), so the group baseline has real work to do.

Run it:

consoleCopypython3 main.py

You will see per-iteration stats (group reward mean, format violations, KL to reference) and a final evaluation where the greedy policy solves held-out prompts. The full math derivation lives in docs/en.md; the trainer is ~200 lines and every GRPO constant in the doc maps to a named constant in the code.

Verify It

code/tests/test_main.py locks down the pieces that make GRPO GRPO:

• the verifier respects operator precedence and rejects malformed expressions;
• group advantages are zero-mean and exactly zero when all rewards tie;
• the clipped ratio stays inside [1-\epsilon, 1+\epsilon] for extreme importance ratios;
• the k_3 estimator is non-negative and zero for identical policies;
• the length reward pays short-correct more than long-correct, and punishes long-wrong hardest;
• seeded end-to-end training raises greedy success rate from cold start.

Why This Matters for Engineers

• Choose RLVR whenever a checker exists. Math answers, code, structured output (JSON schema validation!), retrieval faithfulness against a labeled corpus. Reserve learned reward models for open-ended quality, and expect to red-team them forever.
• The verifier is part of your attack surface. Design it before scaling: require the reasoning format, keep partial credit out until you need it, and audit for "guessing beats reasoning" equilibria.
• Group size is your value-function resolution. G=8 is common; hard prompt distributions want larger groups or curriculum filtering (drop prompts with all-zero advantages — GRPO does this implicitly).
• The KL anchor is a cost dial. It is the difference between "RL improves the model" and "RL produces a reward-maximizing degenerate dialect".
• This exact loop is what runs inside R1-class pipelines; the LLM is bigger, the rollouts come from vLLM, but the objective is the one you just trained by hand.

Hints

• If group rewards are all identical, the advantage is zero by definition — check that your training mix includes prompts near the model's capability frontier.
• A negative reward mean with zero format violations usually means the policy found a valid expression family that never hits targets: raise \beta or reseed the prompt mix.
• Length collapse shows up as success rate holding while tokens per response inflate; the k1.5 term exists precisely for that.

Key Terms