Three ways to change a model
Post-training is everything that happens to a language model after pre-training on a large corpus. For enterprise work it comes down to three families of methods, each answering a different question:
- Supervised fine-tuning (SFT) answers: what does a good output look like? The model imitates demonstrations.
- Preference optimization (DPO and relatives) answers: which of these outputs is better? The model learns from comparisons.
- Reinforcement learning (RLHF, GRPO, PPO) answers: how do I get a higher score? The model explores and is rewarded.
They are not competitors so much as stages. Nearly every strong model we ship goes through SFT first, then one or more rounds of preference optimization or RL.
Supervised fine-tuning (SFT)
SFT trains the model with ordinary next-token prediction on curated input–output pairs. It is simple, stable and data-efficient, and it does most of the heavy lifting for format, vocabulary and procedure.
Data: 500–20,000 demonstrations, depending on task difficulty. Quality dominates quantity: a demonstration with a subtle error teaches that error confidently.
Strengths: predictable, cheap, easy to debug. Excellent at making outputs consistent.
Limits: the model can only become as good as the demonstrations, and it learns to imitate the average of them. It has no signal about which of two acceptable outputs is preferable, and it never sees its own mistakes.
Read more on our supervised fine-tuning service page.
Direct Preference Optimization (DPO)
DPO trains on pairs: for a given prompt, a chosen response and a rejected one. The loss pushes the model to raise the relative probability of the chosen response compared to a frozen reference model. Mathematically it optimizes the same objective as RLHF under certain assumptions, but without training a separate reward model or running a reinforcement-learning loop.
Data: 2,000–10,000 preference pairs typically produce clear gains. Pairs are most informative when both responses are plausible and differ on what you care about.
Variants: IPO adds regularization against overfitting; KTO works from binary thumbs-up/down labels instead of pairs, which is convenient when you have production feedback; ORPO folds preference learning into SFT in a single stage; SimPO removes the reference model.
Strengths: stable, cheap (roughly the cost of an SFT run), and remarkably effective at improving judgment, tone and the suppression of specific failure modes.
Limits: it is offline — the model learns from fixed responses rather than its own current behavior — so gains flatten after a few rounds unless you regenerate pairs from the latest checkpoint (iterative or “online” DPO).
RLHF with a reward model
Classic RLHF trains a reward model on human preference data, then uses reinforcement learning (usually PPO) to optimize the language model against that reward, with a penalty for drifting too far from the starting model.
Data: tens of thousands of comparisons for a robust reward model, plus a large pool of prompts.
Strengths: on-policy — the model is evaluated on what it currently generates, so it can keep improving where offline methods stall. The reward model is also a reusable asset for ranking and monitoring.
Limits: engineering-heavy, more expensive and prone to reward hacking, where the model finds outputs that score well but aren’t actually better — verbose answers, confident hedging, gaming the rubric. It needs careful monitoring.
RL with verifiable rewards (GRPO and friends)
When outputs can be checked automatically — code that compiles and passes tests, SQL that returns the right rows, a tool call that succeeds, a classification that matches a known label — you can skip the learned reward model and reward the model directly from a verifier. Group Relative Policy Optimization (GRPO) popularized an efficient way to do this: sample several responses per prompt, score them, and push the model toward the better-than-average ones.
Data: prompts and verifiers rather than labels. A few thousand representative tasks can be enough.
Strengths: can take a model well beyond its demonstrations, especially on multi-step reasoning and agentic tool use. The reward is grounded in reality rather than in a model’s opinion.
Limits: only as good as the verifier. Weak verifiers get exploited — a test suite with gaps will be found and used. Compute-hungry, since every step requires sampling.
Comparison at a glance
| Method | Learns from | Typical data | Relative cost | Best for |
|---|---|---|---|---|
| SFT | Demonstrations | 0.5k–20k examples | 1× | Format, vocabulary, procedure |
| DPO / KTO | Comparisons or ratings | 2k–10k pairs | 1–1.5× | Judgment, tone, suppressing failures |
| RLHF (PPO) | Reward model | 20k+ comparisons | 3–6× | Nuanced quality at scale |
| GRPO / RLVR | Verifiers | 2k+ prompts with checks | 3–10× | Reasoning, code, tool use, agents |
A worked example: underwriting submissions
A commercial insurer wanted a model to read broker submissions — emails, attachments, loss runs — and produce a structured risk summary for underwriters, with a recommendation to quote, refer or decline.
SFT stage. We mined 7,400 historical submissions where an underwriter had written a summary and recorded a decision, cleaned them, and had senior underwriters correct 1,100 that were incomplete. SFT on a 14B base brought schema validity to 99.8% and field accuracy from 61% to 87%. But the recommendations were bland: the model referred almost everything, because referral was the most common label in the training data.
DPO stage. We sampled four responses per submission from the SFT model and asked underwriters to pick the best and worst. 4,800 pairs later, DPO taught the model what good judgment looked like. The referral rate dropped from 71% to 38%, matching senior underwriter behavior, and blind reviewers preferred the DPO model’s summaries 68% of the time.
RL stage — skipped. There was no reliable verifier for “correct underwriting decision” on the time scale of training (the true outcome is a loss ratio years later), and the eval showed the DPO model already within the range of underwriter disagreement. Running RL would have cost more and risked optimizing a proxy. Knowing when to stop is part of the method.
Data requirements in practice
Published papers often report results with large, clean datasets. Enterprise reality is different, and it changes which methods are practical:
- Demonstrations are expensive when experts must write them from scratch. Mining existing outputs — historical decisions, edited drafts, resolved tickets — is usually the cheapest source of SFT data.
- Comparisons are cheap. Experts can judge two outputs far faster than they can write one. That makes DPO the most cost-effective way to inject expert judgment.
- Production feedback is free but noisy. Thumbs up/down, accepted suggestions and edits are natural inputs for KTO-style training, provided you filter for rater reliability.
- Verifiers are an engineering investment. Writing a robust checker for SQL correctness or tool-call validity takes days to weeks, but then produces unlimited reward signal.
How we sequence them
Our default recipe for an enterprise task:
- SFT on curated demonstrations to establish format and baseline competence.
- Error analysis on the held-out set. Categorize every failure.
- DPO on pairs targeting the dominant failure categories, using the SFT model’s own outputs as rejected examples and expert corrections as chosen ones.
- RL with verifiers if the task has checkable outcomes and the eval shows headroom — common for agents, code and structured extraction.
- Regression check after every stage against general capabilities and safety.
Many tasks are done after step 3. Reaching for RL first is a common and expensive mistake; reaching for it never leaves real gains on the table for agentic and reasoning workloads.