The fine-tuning playbook, written down.
The process behind 214 production models. Every phase has a deliverable you keep and a decision point where you can stop.
Frame
We turn “make the model better” into a contractable target.
- Stakeholder interviews with the people who do the task today
- Task taxonomy and failure-mode inventory from production logs
- A frozen, human-adjudicated held-out set you sign off on
- Baseline scorecard: frontier API, open models, current system
- A written success metric and a ship/no-ship bar
Success-criteria doc · Baseline report
Data
Data is most of the work and most of the result.
- Log and document mining with PII redaction in your environment
- Annotation guidelines written with your domain experts
- Expert labelling and preference collection with inter-rater agreement tracking
- Synthetic data generation filtered by verifiers and judges
- Contamination checks against the held-out set
Versioned datasets · Data card
Train
Short, instrumented cycles, each one decided by the eval.
- Base-model bake-off across 3–5 candidates and sizes
- SFT with hyperparameter sweeps, LoRA vs full-parameter ablations
- Preference tuning (DPO/KTO) or RL with verifiable rewards
- Error analysis after every round, fed back into data
- Everything tracked: configs, seeds, data hashes, metrics
Checkpoints · Ablation log
Prove
Nothing ships on vibes.
- Held-out evaluation with confidence intervals
- Side-by-side blind human review against the incumbent
- Red-teaming for jailbreaks, leakage and harmful output
- Regression suite of hand-written assertions
- Model card and risk memo for your model-risk or AI governance board
Sign-off report · Model card
Ship & run
A model is a living system.
- Quantization (FP8, INT4 AWQ/GPTQ) with accuracy checks
- Serving on vLLM, SGLang or TensorRT-LLM in your cloud
- Shadow traffic, canary release, automated rollback
- Drift and quality monitoring tied back to the eval
- Scheduled retraining on fresh production data
Serving stack · Runbook
The eval comes first. We build the measuring instrument before the thing being measured. If we can’t agree what “better” means, we don’t train.
Small beats big when it’s trained right. We start from the smallest model that could plausibly work and only scale up when the eval demands it.
Your people are the moat. The labels and preferences from your experts are the asset. Our job is to turn their judgment into weights.
Leave them stronger. Every engagement ends with your team able to run the next cycle. We measure ourselves by how little you need us afterwards.
Working together
What do you need from us to start?
A named task owner, access to representative examples (even a few hundred), 2–4 hours a week from a domain expert during the data phase, and a target environment for deployment.
Who is on the team?
A research lead who owns the outcome, one or two ML engineers, an evaluation engineer and a data lead who runs the annotation program. Every person on your project has trained models that run in production.
How do you report progress?
A weekly written update with the current scorecard, a shared experiment tracker, and a live demo at the end of each phase. No slide decks without numbers.
Bring us the task. We'll bring the model.
A 45-minute technical briefing with a senior research engineer. You leave with a feasibility read, a data plan and a fixed quote — whether or not you hire us.
Book a technical briefing →