What is GRPO in machine learning?

Group Relative Policy Optimization is a reinforcement-learning algorithm for language models that samples several responses per prompt, standardizes their rewards within the group to get advantages, and updates the policy with a PPO-style clipped objective plus a KL penalty, without a learned value function.

Is GRPO better than DPO?

They suit different data. GRPO needs a reward or verifier and generates samples during training; it can exceed the quality of demonstrations on verifiable tasks. DPO needs only fixed preference pairs and is far cheaper. Many pipelines use DPO first and GRPO where outcomes are checkable.

Why do some GRPO prompts give zero gradient?

If every response in a group gets the same reward, all standardized advantages are zero. Prompts the model always solves, or never solves, therefore contribute no learning signal.

GRPO Explained: Group Relative Policy Optimization Math, Example & When to Use It

The idea in one paragraph#

For each prompt, sample a group of $G$ responses from the current policy, score each with a reward (often a programmatic verifier), and push the policy toward responses that scored above the group’s average and away from those below. The group average replaces the learned value function PPO uses as a baseline, saving the memory and compute of a second large network. GRPO was introduced in the DeepSeekMath paper and later popularized by reasoning-model training.

Group-relative advantages#

Given rewards $r_{1}, \dots, r_{G}$ for the $G$ responses to one prompt, each response’s advantage is its standardized reward:

\hat{A}_{i} = \frac{r _{i} - mean ( r _{1} , \dots , r _{G} )}{std ( r _{1} , \dots , r _{G} )}

In the outcome-reward setting, every token of response $i$ receives the same advantage $\hat{A}_{i}$ .

A worked example#

Four SQL queries are sampled for one question; a verifier runs them and returns 1 if the result matches the reference, else 0. Rewards $(1, 0, 0, 1)$ : mean 0.5, standard deviation 0.5, advantages $(+ 1, - 1, - 1, + 1)$ . The two correct queries are reinforced and the two wrong ones suppressed with equal weight.

Now suppose the question is easy and all four are correct: rewards $(1, 1, 1, 1)$ , every advantage is zero (implementations add a small $ϵ$ to the denominator), and the prompt contributes no learning signal. The same happens when all four fail. Only prompts the model sometimes gets right teach it anything.

import torch
def group_advantages(rewards: torch.Tensor, eps: float = 1e-6) -> torch.Tensor:
    # rewards: [num_prompts, G]
    mean = rewards.mean(dim=1, keepdim=True)
    std = rewards.std(dim=1, keepdim=True, unbiased=False)
    return (rewards - mean) / (std + eps)

r = torch.tensor([[1., 0., 0., 1.], [1., 1., 1., 1.]])
print(group_advantages(r))   # [[ 1, -1, -1,  1], [0, 0, 0, 0]]

The objective#

GRPO keeps PPO’s clipped importance ratio and adds a KL penalty to a reference model directly in the loss (rather than in the reward). With $ρ_{i, t} = π_{θ} (o_{i, t} ∣ q, o_{i, < t}) / π_{θ_{old}} (o_{i, t} ∣ q, o_{i, < t})$ :

J (θ) = E \frac{1}{G} i = 1 \sum G \frac{1}{∣ o _{i} ∣} t = 1 \sum ∣ o_{i} ∣ (min (ρ_{i, t} \hat{A}_{i}, clip (ρ_{i, t}, 1 - ε, 1 + ε) \hat{A}_{i}) - β D_{i, t})

where the per-token KL is estimated with the non-negative, unbiased estimator

D_{i, t} = \frac{π _{ref} ( o _{i, t} ∣ \cdot )}{π _{θ} ( o _{i, t} ∣ \cdot )} - lo g \frac{π _{ref} ( o _{i, t} ∣ \cdot )}{π _{θ} ( o _{i, t} ∣ \cdot )} - 1

The clipping (typically $ε = 0.2$ ) stops any single update from moving the policy too far from the one that generated the samples.

Known biases and fixes#

Two normalizations in the objective have side effects that Liu et al. analyse in detail:

Length normalization $1/∣ o_{i} ∣$ : for a negative advantage, spreading the penalty over more tokens makes each token’s penalty smaller, so long wrong answers are punished less than short wrong ones. Over training this can inflate response length, especially on incorrect outputs.
Standard-deviation normalization: prompts where the group’s rewards barely vary (nearly all right or nearly all wrong) get their small differences amplified, weighting very easy and very hard questions more heavily.

Their proposed variant, Dr. GRPO, removes both normalizations. For enterprise tasks where response length matters (tool calls, structured outputs, support replies), we monitor length on held-out prompts and prefer the unbiased form.

GRPO vs PPO vs DPO#

	PPO (RLHF)	GRPO	DPO
Learns from	reward model scores on fresh samples	rewards on groups of fresh samples	fixed preference pairs
Extra networks in memory	reward model + value model	reward model or verifier only	reference model only
Generation during training	yes	yes, G per prompt	no
Relative cost	highest	high	low
Best for	nuanced learned rewards	verifiable outcomes	expert comparisons

On compute: a GRPO step with $G = 8$ generates eight completions per prompt, and generation runs at low hardware utilisation, which is why RL stages dominate project compute (compute model).

Using GRPO on enterprise tasks#

The quality of a GRPO run is bounded by its verifier. Verifiers we build most often:

Task	Verifier	Exploit to guard against
Text-to-SQL	execute against a snapshot, compare result sets	queries that return the right rows by accident (e.g. LIMIT tricks)
Tool-calling agent	schema validation + simulated tool execution	valid calls that skip a required step
Structured extraction	field-level match to labelled targets	copying spans that look right without normalizing
Classification with rationale	label match + rationale consistency judge	rationales that contradict the label

Two operational rules follow from the zero-advantage property. Curate a prompt set the SFT model solves sometimes but not always — filter out prompts with group accuracy 0 or 1 after a few epochs. And size $G$ to the pass rate: if a prompt succeeds 10% of the time, a group of 4 has a 66% chance of containing no success at all ( $0. 9^{4} \approx 0.66$ ); a group of 16 brings that to 19%.

References#

Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
Liu, Z. et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783.
Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155.

GRPO explained: group-relative policy optimization, with the math