Why generic benchmarks don’t help
Public benchmarks measure broad capabilities on public data. A fine-tuned model is built for a specific task on private data, and it is supposed to get worse at some generic things in exchange for being much better at yours. Leaderboard scores won’t tell you whether it extracts the right policy number from a scanned endorsement, or whether it escalates the angry customer who mentions a lawyer.
You need a task-specific evaluation: your inputs, your definition of correct, your failure modes. Build it before training, freeze it, and use it for every decision.
Building the golden set
A golden set is a collection of inputs with agreed-upon correct outputs (or scoring rules), never used in training.
- Sample from production. Draw from real traffic, stratified by segment, input type and difficulty. Over-sample rare but costly categories so they are measurable.
- Adjudicate. Have two experts label independently; resolve disagreements with a third. Track inter-rater agreement — if experts agree only 80% of the time, no model will reliably score above that.
- Write the rubric down. Every ambiguity you resolve becomes a rule. The rubric later becomes the judge prompt.
- Freeze and version it. Changing the test set mid-project makes results incomparable.
- Keep a hidden slice. A portion nobody on the training team sees, used only for final sign-off.
Choosing metrics
Use the most objective metric the task allows, and add judgment-based metrics only where needed.
| Task type | Primary metric | Notes |
|---|---|---|
| Classification, routing | Accuracy, per-class precision/recall, macro-F1 | Report the confusion matrix; cost-weight critical errors |
| Extraction | Field-level exact match, normalized match | Separate “missing” from “wrong” errors |
| Code, SQL, tool calls | Execution success, result correctness | Run it; don’t compare strings |
| Summaries, drafts | Rubric scores via calibrated judge + human review | Check faithfulness to source separately from style |
| Agents | Task completion, steps, escalation correctness | Evaluate whole trajectories, not single turns |
Calibrating LLM-as-judge
Automated judges make evaluation fast enough to run on every checkpoint, but an uncalibrated judge is a random number generator with good grammar. To calibrate:
- Have experts score 200–400 outputs using the written rubric.
- Run the judge on the same outputs.
- Measure agreement: Cohen’s kappa for categorical scores, Spearman correlation for scales, and inspect the confusion matrix.
- Revise the judge prompt — usually by adding rubric detail and worked examples — until agreement approaches human–human agreement.
- Re-check on a fresh sample each quarter, and whenever the judge model changes.
Watch for known judge biases: preference for longer answers, for the first option in a pair, and for outputs from its own model family. Randomize order and control for length.
Sample sizes and statistics
Small evaluation sets produce confident-looking numbers that don’t replicate. As a guide, to detect a 3-point accuracy difference around 85% with reasonable confidence you need roughly 1,000–1,500 examples. To report per-segment results, each segment needs its own adequate sample.
Always report a confidence interval — bootstrap resampling is simple and robust — and use paired comparisons when evaluating two models on the same inputs. A model that wins by 1.2 points with an interval of ±2.5 has not won.
Regression suites and release gates
Beyond the headline metric, maintain a regression suite: hand-written test cases for behaviors that must never break — refusing to give dosage advice, never inventing a policy number, always returning valid JSON. These are assertions, not statistics; a single failure blocks release.
Wire both into CI as release gates: a candidate ships only if it beats the incumbent on the primary metric (outside the confidence interval), passes every regression assertion, and stays within latency and cost budgets. The same gates protect you when a prompt changes, a retrieval index is rebuilt or a vendor updates a model underneath you.
Human review and red-teaming
Before first launch, run a blind side-by-side review: experts compare the candidate and the incumbent (or the current human process) without knowing which is which. This catches quality issues automated metrics miss and builds stakeholder trust that no dashboard can.
Then red-team: deliberately try to make the model leak data, follow injected instructions, produce harmful advice or behave inconsistently across demographic groups. Document what you tried and what you found. For regulated industries, package everything — golden-set design, metrics, judge calibration, red-team results — into a model card your governance process can review.
What a sign-off scorecard looks like
Every model we hand over comes with a one-page scorecard. Decision-makers read this page; engineers read the appendix behind it. A typical example:
| Check | Incumbent | Candidate | Gate |
|---|---|---|---|
| Primary metric (field exact match) | 86.9% ± 1.3 | 94.7% ± 0.9 | Pass |
| Worst segment (handwritten forms) | 71.2% ± 4.1 | 85.0% ± 3.4 | Pass |
| Regression assertions | 409 / 412 | 412 / 412 | Pass |
| Blind human preference | — | 71% preferred | Pass |
| PII leakage probes | 0 / 5,000 | 0 / 5,000 | Pass |
| p95 latency | 4.8 s | 0.41 s | Pass |
| Cost per 1,000 requests | $11.20 | $0.31 | Pass |
Notice the worst-segment row. Aggregate numbers hide the slices where a model fails, and those slices are where incidents come from. Always report the weakest segment next to the headline.
An evaluation maturity model
Most organizations move through four stages. Knowing where you are tells you what to build next.
- Vibes. Engineers try a handful of prompts and eyeball the output. Fine for a demo; dangerous for anything else.
- Spreadsheet. A few hundred examples scored by hand when someone remembers. Better, but slow, inconsistent and rarely rerun.
- Harness. Versioned golden sets, automated metrics and calibrated judges run in CI on every change. This is the minimum for production.
- Loop. Online evaluation on live traffic, human review queues, and confirmed failures flowing back into training and the golden set. This is where fine-tuned models compound.
Our evaluation service typically takes a team from stage one or two to stage three in two to four weeks.
After launch: online evaluation
Production traffic drifts. Sample a small percentage of live requests, score them with your calibrated judges, route a subset to human review, and alert when quality metrics move. Feed confirmed errors back into the next training round. This loop is what keeps a fine-tuned model ahead of the generalist it replaced — see deployment and model ops.