Custom AI models for the enterprise · New York · LondonSecurityResearchContact sales
Finetune the World
Glossary

RLHF (Reinforcement Learning from Human Feedback): definition, math and practical numbers

RLHF trains a reward model on human preference comparisons, then optimizes the language model against that reward with reinforcement learning under a KL penalty.

By Finetune the World Research3 min readPublished

Definition#

RLHF, as popularized by InstructGPT (Ouyang et al. 2022), has three stages: supervised fine-tuning on demonstrations; training a reward model on human comparisons with the Bradley–Terry loss ; and optimizing the policy, typically with PPO, to maximize

The KL term keeps the policy near the SFT model, limiting reward hacking — outputs that exploit flaws in the learned reward. RLHF is on-policy: it learns from the model’s current outputs, which lets it keep improving where offline methods like DPO plateau, at a much higher engineering and compute cost.

See SFT vs DPO vs RLHF.

Related reading

GlossaryAll glossary termsPillar guideLLM post-training: the complete guideResearchOur research

Bring us one task. We’ll show you what a specialist model can do.

A 45-minute briefing with a senior research engineer. You leave with a feasibility read, a data plan and a fixed quote, whether or not you hire us.

Book a briefing →