DPO from first principles, and what β really does with finite data

Start from the RLHF objective#

RLHF as described by Ouyang et al. optimizes a policy $\pi$ to maximize a learned reward while staying close to a reference policy ${\pi }_{\text{ref}}$ (usually the SFT model):

The coefficient $\beta$ prices deviation from the reference: large $\beta$ keeps the policy conservative, small $\beta$ lets it chase reward.

The optimal policy has a closed form#

For each prompt $x$, the objective is maximized (over all distributions) by a Gibbs distribution:

To see this, rewrite the objective for one prompt as $-\beta \mathrm{KL}(\pi \Vert {\pi }^{\ast })+\beta \log Z(x)$. The second term does not depend on $\pi$, and the KL term is maximized at zero when $\pi ={\pi }^{\ast }$. The partition function $Z(x)$ sums over every possible response, which is why this form is not directly usable — until we invert it.

Invert it: the reward is a log-ratio#

Take logs and solve for the reward:

Every reward function corresponds to an optimal policy, and the reward can be read off that policy up to a prompt-dependent constant. This is the sense of the DPO paper’s subtitle: the language model is a reward model.

Plug into Bradley–Terry; the partition function cancels#

Human preferences are modelled with Bradley–Terry: the probability that response $y_w$ is preferred to $y_l$ depends only on the reward difference,

Substituting the log-ratio form, the $\beta \log Z(x)$ terms are identical for both responses and cancel. Replacing ${\pi }^{\ast }$ with the trainable policy ${\pi }_{\theta }$ and maximizing the likelihood of the observed preferences gives the DPO loss:

No reward model, no sampling, no value function: just a classification loss on pairs. Its gradient is informative:

where ${\hat{r}}_{\theta }=\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})$ is the implicit reward. Pairs the model already ranks correctly get small weights; mis-ranked pairs get large ones.

The catch: β only constrains the population optimum#

The derivation guarantees that the population minimizer of the DPO loss is ${\pi }^{\ast }$. A finite dataset is different. Nothing in the loss refers to responses that never appear in a pair, and the loss on observed pairs keeps decreasing as the chosen/rejected log-ratio gap grows. With little data and many steps, the policy can move much further from ${\pi }_{\text{ref}}$ than $\beta$ intends. Azar et al. make a related theoretical argument and propose IPO as a remedy.

A simulation where we know the right answer#

To measure this directly we built a problem with a known optimum: 64 prompts, 8 candidate responses each, true rewards $r\sim \mathcal{N}(0,1)$, and a random reference policy. Preference pairs are sampled from ${\pi }_{\text{ref}}$ and labelled by the Bradley–Terry model. The policy is a free table of logits (the most flexible possible model) trained with Adam on the DPO loss for 3,000 steps. Because everything is tabular, we can compute ${\pi }^{\ast }$ exactly and measure how close DPO gets.

What we found#

1. With little data, β barely changes how far the policy moves#

At 256 pairs, the realized KL from the reference ranged only from 1.48 to 1.83 across a twentyfold range of $\beta$, while the target KL ranged from 0.31 to 2.39. At $\beta =1$ the policy moved 4.8× further than intended.

2. Large β can overshoot the target#

At $\beta =1$ with 2,048 pairs, DPO reached a true reward of 0.677 — higher than the optimum’s 0.650 — at a KL of 1.75, 5.6× the regularized optimum. Higher reward sounds good, but with a learned or noisy reward this is the regime where reward hacking lives, and you asked for a conservative policy.

3. Data size dominates#

At $\beta =0.1$, the distance to the optimum fell from 8.84 nats at 256 pairs to 0.72 at 16,384. At 256 pairs, 48% of prompt–response combinations never appeared in any pair, and their probabilities were determined by normalization alone.

Training dynamics: the loss keeps falling, the policy keeps moving#

The trajectories for 2,048 pairs show why step count matters as much as $\beta$. The DPO loss on the training pairs falls monotonically, and the KL from the reference keeps climbing well after the loss has flattened:

Early stopping on a held-out KL or reward-margin budget is therefore a more reliable regularizer than $\beta$ alone in data-limited settings.

What to monitor in a real DPO run#

In a real model you cannot compute ${\pi }^{\ast }$, but you can track the quantities that revealed the problem here, on held-out prompts:

import torch
@torch.no_grad()
def dpo_monitors(policy_logps_w, policy_logps_l, ref_logps_w, ref_logps_l, beta):
    """Summed log-probs of chosen (w) and rejected (l) responses, shape [batch]."""
    r_w = beta * (policy_logps_w - ref_logps_w)      # implicit reward, chosen
    r_l = beta * (policy_logps_l - ref_logps_l)      # implicit reward, rejected
    return {
        "reward_margin": (r_w - r_l).mean().item(),  # grows without bound when overfitting
        "reward_acc":    (r_w > r_l).float().mean().item(),
        "chosen_logp_shift": (policy_logps_w - ref_logps_w).mean().item(),  # < 0 is a red flag
    }

A falling chosen_logp_shift — the model making even the preferred responses less likely, while widening the gap — is a frequently reported DPO failure pattern and a cue to stop or add an SFT term on the chosen responses.

Variants that change the regularization#

IPO (Azar et al.) replaces the logistic loss with a squared loss that targets a finite margin instead of pushing it to infinity:

Because the optimum is a fixed margin, IPO cannot drive log-ratios apart indefinitely on deterministic preferences, which directly addresses the overshoot in finding 2. Our simulation code supports swapping the loss in one line if you want to reproduce the comparison.

Practical implications#

• Don’t rely on β for safety margins. Monitor the realized KL (or the implicit reward margins) on held-out prompts and stop early when it exceeds your budget.
• Prefer more, broader pairs over more epochs. One or two epochs over diverse prompts generalize better than many passes over few.
• Regenerate pairs from the current policy (iterative DPO) so the data covers what the model actually produces.
• Consider IPO or a KL early-stopping rule when preference data is small relative to the response space.

Our preference-tuning engagements use realized-KL budgets as a release gate for exactly this reason.

Limitations#

A tabular policy is far more flexible than a neural network, which generalizes across responses; real models will not leave unseen responses untouched in the same way. Rewards are synthetic and the Bradley–Terry model is exactly correct here, which is never true of human raters. The results show a mechanism, not a calibrated prediction for any specific language model.

References#

1. Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arXiv:2305.18290.
2. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155.
3. Bradley, R. A. and Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39(3/4), 324–345.
4. Azar, M. G. et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv:2310.12036.