LLM-as-judge: definition, math and practical numbers

Definition#

An LLM judge scores outputs pass/fail or on a scale, or compares two outputs pairwise. Zheng et al. (2023) documented systematic judge biases, including preference for the first-presented answer, for longer answers and for outputs resembling the judge’s own.

Treat a judge like a diagnostic test with sensitivity $\text{se}$ and specificity $\text{sp}$. The pass rate it reports is $q=\text{se}p+(1-\text{sp})(1-p)$, so measured differences between models shrink by Youden’s $J=\text{se}+\text{sp}-1$. In our simulation, a judge with $J=0.62$ reported a real 5-point gain as 3.1 points, and the naive interval covered the truth only 72% of the time. The Rogan–Gladen estimator corrects this using a human-labelled calibration set (full analysis).