How many evaluation examples do you need to compare two LLMs?

The short answer#

At a baseline accuracy of 85%, two-sided $\alpha =0.05$ and 80% power, the examples needed to detect an improvement of $\Delta$ are:

Two lessons. First, small improvements are expensive to prove: a 1-point gain needs around 20,000 examples, which is why claims of “+1.2 points” on a few hundred items are noise. Second, evaluating both models on the same items and analysing the pairs cuts the requirement substantially — but only to the extent that the two models get the same items right and wrong.

The analytic baseline#

Treating the two models’ accuracies $p_A$ and $p_B=p_A+\Delta$ as independent proportions, the required examples per model for a two-sided test are:

With $z_{0.975}=1.96$, $z_{0.8}=0.84$, $p_A=0.85$ and $\Delta =0.03$, that is 2,033. The $1/{\Delta }^2$ dependence is the whole story of evaluation budgets: halving the effect you want to detect quadruples the data.

Why pairing helps#

In practice both models are scored on the same items, and items vary in difficulty. A hard item is likely wrong for both models; an easy one right for both. Those concordant items carry no information about which model is better. McNemar’s test uses only the discordant pairs — $n_{10}$ items where only A is right and $n_{01}$ where only B is right:

The variance of the paired difference is $\mathrm{Var}({\hat{p}}_B-{\hat{p}}_A)=\frac{1}{n}\left[p_A(1-p_A)+p_B(1-p_B)-2\mathrm{Cov}\right]$. The more the models’ errors coincide, the larger the covariance and the fewer examples you need.

Simulation design#

To quantify the effect we simulated items with latent difficulty. Item $i$ has logit $z_i\sim \mathcal{N}(\mu ,{\sigma }^2)$; model A answers correctly with probability $\sigma (z_i)$ and model B with $\sigma (z_i+\delta )$, with $\mu$ and $\delta$ solved numerically so that mean accuracies are exactly 85% and $85\%+\Delta$. The spread $\sigma$ controls how much the models share their errors: $\sigma =0$ means independent errors; $\sigma =3$ means a strong easy/hard split typical of real benchmarks. For each design and $n$ we ran 4,000 simulated evaluations and counted how often the test rejected at $\alpha =0.05$.

With independent errors ($\sigma =0$) pairing gains nothing — the paired and unpaired requirements match within simulation resolution. At $\sigma =3$, a 3-point improvement needs 1,165 paired items versus 2,046 unpaired, a 43% reduction.

What to do with this#

• Decide the minimum improvement worth detecting before building the set. If a 2-point gain would change your decision, you need thousands of items, not hundreds.
• Always evaluate candidates on identical items and analyse paired differences. Report the paired bootstrap interval of the difference, not two separate intervals.
• Measure your own error overlap. The fraction of items on which two models agree tells you which row of the table you are in. On extraction tasks we commonly see 85–95% agreement, firmly in the “strongly shared” regime.
• Stratify. Per-segment claims need per-segment sample sizes; a 1,500-item set split into ten segments supports almost no segment-level conclusions.

The paired bootstrap, in code#

McNemar’s test answers “is there a difference?” For reporting, you want an interval for the size of the difference. The paired bootstrap resamples items, keeping each item’s two scores together:

import numpy as np
def paired_bootstrap(a: np.ndarray, b: np.ndarray, reps: int = 10_000, seed: int = 0):
    """a, b: per-item scores (0/1 or graded) for the same items. Returns (diff, lo, hi)."""
    rng = np.random.default_rng(seed)
    n = len(a); d = b - a
    idx = rng.integers(0, n, size=(reps, n))
    boot = d[idx].mean(axis=1)
    return d.mean(), *np.percentile(boot, [2.5, 97.5])

# Example: 1,500 items, candidate right on 52 items the incumbent missed, wrong on 18 it got
a = np.r_[np.ones(1300), np.ones(18), np.zeros(52), np.zeros(130)]
b = np.r_[np.ones(1300), np.zeros(18), np.ones(52), np.zeros(130)]
print(paired_bootstrap(a, b))   # ≈ (0.023, 0.012, 0.034)

Only the 70 discordant items move the estimate; the 1,430 items both systems agree on contribute nothing but stability. That is the pairing advantage in miniature.

Segment-level claims need segment-level samples#

A 1,500-item set may detect a 3-point overall gain, but split into ten equal segments it has 150 items per segment, enough to detect only differences of roughly 10–12 points within any one of them. If the business cares about the worst segment — handwritten forms, a minority language, a rare product line — over-sample it deliberately and report it separately. The relationship is the same $1/{\Delta }^2$ law applied per segment:

Many checkpoints, many chances to fool yourself#

Fine-tuning projects compare dozens of checkpoints and configurations against the same evaluation set. Each comparison at $\alpha =0.05$ carries a 5% false-positive risk; across $k$ independent comparisons the chance of at least one spurious “win” is

which is 40% at $k=10$ and 64% at $k=20$. Two defences: hold out a confirmation slice that is used exactly once, on the final candidate; and, where you must make many formal comparisons, tighten the threshold (Bonferroni: $\alpha /k$). Selection on the development set then becomes harmless, because the claim you report comes from data the selection never saw.

Limitations#

Binary correctness only; graded or rubric scores have different variance. The logit-normal difficulty model is a convenient assumption, not an empirical fit, and simulated required sizes were found by a geometric search with about 8% resolution. Multiple comparisons across many candidate checkpoints inflate false positives; correct for them or hold out a final confirmation set.

References#

1. McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157.