DPO (Direct Preference Optimization): definition, math and practical numbers

Definition#

Direct Preference Optimization (Rafailov et al. 2023) trains on triples $(x,y_w,y_l)$: a prompt, a preferred response and a rejected one. Its loss is a logistic loss on the difference of log-probability ratios against a frozen reference model:

It follows from the KL-regularized RLHF objective: the optimal policy satisfies $r=\beta \log ({\pi }^{\ast }/{\pi }_{\text{ref}})+\beta \log Z$, and under the Bradley–Terry preference model the intractable $Z$ cancels. Full derivation: DPO from first principles.

The β caveat#

$\beta$ nominally sets how far the policy may move from the reference. In our tabular simulation with 256 preference pairs, the realized KL varied only from 1.48 to 1.83 across a twentyfold range of β. Monitor divergence directly.