Definition#
GRPO (Shao et al. 2024, introduced with DeepSeekMath) samples responses per prompt, scores them, and assigns each the standardized advantage
which then weights a PPO-style clipped policy-gradient objective with a KL penalty to a reference model. Because the baseline comes from sibling samples, no value network is trained, which saves memory and compute relative to PPO.
Key property#
If every response in a group gets the same reward, all advantages are zero and the prompt contributes nothing. Training sets should focus on prompts the model solves sometimes but not always. Group size should reflect the pass rate: at a 10% success rate, a group of 4 contains no success 66% of the time.