Your LLM judge is shrinking your results. A 1978 estimator fixes it.

The problem#

LLM-as-judge evaluation replaces expensive human grading with a model that labels each output pass or fail. Judges are useful and cheap, and they are wrong some of the time. Treat a judge like a diagnostic test with two error rates: its sensitivity $\text{se}=P(\text{pass}\mid \text{good})$ and its specificity $\text{sp}=P(\text{fail}\mid \text{bad})$. If the true pass rate is $p$, the rate the judge reports is:

Lenient judges — high sensitivity, low specificity — are common in practice: they pass most good outputs and a good share of bad ones.

Differences shrink by Youden’s J#

Compare two models with true pass rates $p_A$ and $p_B$, graded by the same judge. Subtracting the two expressions for $q$:

The measured difference is the true difference multiplied by Youden’s $J$, which is below 1 for any imperfect judge. A judge with 92% sensitivity and 70% specificity has $J=0.62$: a genuine 5-point improvement is reported as 3.1 points. Worse, the standard confidence interval is built around the shrunken estimate, so it confidently excludes the truth.

The Rogan–Gladen correction#

Solving the first equation for $p$ gives the estimator Rogan and Gladen proposed in 1978 for screening tests:

You need estimates of the judge’s sensitivity and specificity, which come from a calibration set: items with human labels that you also run through the judge. The uncertainty in $\widehat{\text{se}}$ and $\widehat{\text{sp}}$ must be carried into the final interval, which we do by bootstrapping the evaluation items and calibration items together.

Simulation: bias and coverage#

Two models with true pass rates 80% and 85% (a 5-point gap), 1,500 evaluation items each, and a calibration set of 300 human-labelled items (150 good, 150 bad). For three judges we ran 600 simulated evaluations and recorded the average measured difference and how often the 95% interval contained the true 5 points:

With a good judge ($J=0.9$) the bias is small and the raw interval is fine. With the lenient judge, the raw estimate is 3.1 points instead of 5, and its interval covers the truth only 72% of the time. The corrected estimator is unbiased (5.0 points) with 93% coverage. The price is variance: its standard deviation is 1.7× larger, because dividing by a small $J$ amplifies noise. You trade a confident wrong answer for an honest, wider one.

Simulation: when the judge prefers one model#

The correction assumes the judge errs the same way on both models. That fails when the judge is more lenient toward one model’s style — longer answers, a familiar tone, its own model family, all documented judge biases. We simulated model A at 80% and model B at 83% with a judge that has 90% sensitivity for both, but specificity 0.7 on A’s failures and 0.85 on B’s:

The better model now looks worse. Across 2,000 simulated evaluations of 2,000 items, the judge ranked A above B 72% of the time. No amount of extra evaluation data fixes this; only per-model calibration does.

A practical recipe#

1. Build a calibration set of 200–400 outputs labelled by your domain experts, balanced between good and bad, drawn from each model you compare.
2. Estimate sensitivity and specificity per model. If they differ materially between models, the judge is not neutral; fix the rubric or switch judges before comparing.
3. Report corrected pass rates and differences with bootstrap intervals that resample both the evaluation and calibration sets.
4. Watch $J$. Below about 0.5 the corrected estimates become too noisy to be useful; invest in a better judge or more human labels.
5. Recalibrate whenever the judge model, its prompt or the candidate models change.

This complements the sample-size analysis in how many evaluation examples you need: judge error effectively multiplies the required sample size by roughly $1/J^2$.

How large a calibration set?#

The corrected estimate’s uncertainty comes from three sources: the evaluation sample, and the estimates of sensitivity and specificity. A first-order (delta-method) approximation for one model’s corrected pass rate is

where $n$ is the number of evaluation items and $m_{+}$, $m_{-}$ the numbers of truly good and truly bad calibration items. Two consequences follow. Everything is divided by $J^2$: halving a judge’s Youden index quadruples the variance. And the calibration terms do not shrink as you add evaluation items, so once $n$ is large, more human labels for calibration are worth more than more evaluation items. With a lenient judge (se 0.92, sp 0.70), $p\approx 0.85$ and 1,500 evaluation items, moving from 150 to 600 calibration items per class cuts the standard error of the corrected rate by about a third in this approximation.

Pairwise judges have their own bias#

Many teams use pairwise judging (“which response is better?”) instead of pass/fail. The same logic applies, with one well-documented addition: position bias, where the judge favours whichever answer it sees first. Zheng et al. measured it across several judge models. The standard mitigation is to run every comparison twice with the order swapped and count a win only when the judge agrees with itself; inconsistent pairs are scored as ties. The fraction of inconsistent pairs is itself a useful judge-quality metric — above 20–25%, the judge is not discriminating reliably on your task.

The correction in code#

import numpy as np
def rogan_gladen(q, se, sp):
    return np.clip((q + sp - 1) / (se + sp - 1), 0, 1)

def corrected_diff(judged_a, judged_b, calib_good, calib_bad, reps=5000, seed=0):
    """judged_*: 0/1 judge verdicts on the eval set for each model.
    calib_good / calib_bad: judge verdicts on human-labelled good / bad items."""
    rng = np.random.default_rng(seed); out = []
    for _ in range(reps):
        a = judged_a[rng.integers(0, len(judged_a), len(judged_a))].mean()
        b = judged_b[rng.integers(0, len(judged_b), len(judged_b))].mean()
        se = calib_good[rng.integers(0, len(calib_good), len(calib_good))].mean()
        sp = 1 - calib_bad[rng.integers(0, len(calib_bad), len(calib_bad))].mean()
        out.append(rogan_gladen(b, se, sp) - rogan_gladen(a, se, sp))
    se, sp = calib_good.mean(), 1 - calib_bad.mean()
    point = rogan_gladen(judged_b.mean(), se, sp) - rogan_gladen(judged_a.mean(), se, sp)
    return point, *np.percentile(out, [2.5, 97.5])

If the calibration items differ by model (as they should when the judge may treat models differently), estimate se and sp separately per model and use each model’s own values in its correction.

Limitations#

Binary pass/fail labels only; graded scores need a misclassification matrix rather than two rates. We assume judge errors are independent across items given the true label, and that calibration items are representative of the evaluation distribution. Real judge errors cluster on ambiguous items, which a stratified calibration set can partly address.

References#

1. Rogan, W. J. and Gladen, B. (1978). Estimating prevalence from the results of a screening test. American Journal of Epidemiology 107(1), 71–76.
2. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685.