LLM Evaluation Plan Template: Golden Set, Metrics, Judges, Release Gates

The template#

EVALUATION PLAN — [model / task name]          Version [1.0]   Frozen on [date]

1. Decision this evaluation supports
   Ship candidate X to replace [incumbent] for [task] if the release gates in §7 pass.

2. Golden set
   Source:            [production logs, date range, channels]
   Size:              [n] items  (justified in §4)
   Stratification:    [segments and target counts per segment]
   Unseen-format slice: [n] items from [channels/templates absent from training data]
   Labelling:         two independent experts; third adjudicates; agreement reported
   Hidden slice:      [n] items never seen by the development team

3. Metrics
   Primary:           [e.g. field-level exact match]
   Secondary:         [per-field accuracy, JSON validity, latency p50/p95, cost/1k]
   Worst-segment:     [primary metric on the weakest segment]

4. Sample size
   Smallest improvement worth detecting: Δ = [3] pts at baseline [85]%
   Design: paired (same items for all candidates); expected agreement [~90]%
   Required n at 80% power, α = 0.05: [~1,200–2,000]

5. Automated judges (if any)
   Judge model and prompt version: [..]
   Calibration set: [300] human-labelled items per candidate model
   Reported: sensitivity, specificity, Cohen's κ vs experts
   Correction: Rogan–Gladen applied to pass rates; bootstrap CI includes calibration

6. Regression suite
   [n] hand-written assertions that must always pass, e.g.
   - never outputs a policy number not present in the input
   - always returns valid JSON for schema v3
   - refuses requests for [prohibited content]

7. Release gates
   [ ] Primary metric ≥ [target]
   [ ] Paired 95% CI of (candidate − incumbent) excludes 0
   [ ] Worst segment ≥ [floor]
   [ ] All regression assertions pass
   [ ] Unseen-format slice ≥ [floor]
   [ ] Blind expert review: candidate preferred or tied in ≥ [60]% of [200] pairs
   [ ] Latency and cost within budget

Notes on each section#

Golden set#

Sample from production, not from the training distribution, and include an unseen-format slice. In our LoRA experiment every configuration from rank 4 up scored identically in-distribution and differed by several points on unseen formats; without that slice the choice would have been arbitrary.

Sample size#

For independent proportions, the required items per model are approximately

n \approx \frac{( z _{0.975} + z _{0.8} ) ^{2} [ p ( 1 - p ) + p ^{'} ( 1 - p ^{'} )]}{Δ ^{2}}

which is about 2,000 for a 3-point gain at 85%. Paired designs need fewer when the models’ errors overlap; our simulation gives the numbers.

Judges#

An uncalibrated judge shrinks differences by its Youden’s J and can flip rankings if it is more lenient to one model’s style. Calibrate per candidate model (details).

Freezing#

Once approved, the plan and the golden set are versioned and frozen. Changing them mid-project makes results incomparable and invites optimism bias.

LLM evaluation plan template