What distillation is
Knowledge distillation trains a smaller student model to reproduce the behavior of a larger teacher. In classic machine learning the student learns from the teacher’s full probability distributions (“soft labels”). In enterprise LLM work it more often means generating a large, high-quality dataset of teacher outputs on your task and fine-tuning the student on them — sometimes called sequence-level distillation.
The key insight is that a student doesn’t need the teacher’s general intelligence; it needs the teacher’s behavior on one distribution of inputs. That narrow target is why a 7B student can match a frontier model on a specific task while costing a small fraction to run.
When distillation is the right move
- Your prototype works on a frontier model, and the remaining problem is cost, latency or privacy, not quality.
- The task is well-defined and high-volume: extraction, classification, summarization to a template, routing, constrained generation, tool calling.
- You can generate or collect representative inputs at scale.
- You need to run inside your network, on-prem or on device.
Distillation is the wrong move if the frontier model itself isn’t good enough yet. A student inherits its teacher’s ceiling — fix quality first, then distill. (A fine-tuned student can surpass its teacher, but that comes from the preference-tuning stage, not the distillation itself.)
Step 1: Build the best possible teacher
The teacher is a system, not just a model: the strongest model you can access, plus the best prompt, retrieval, tools and reasoning budget you can give it. Because teacher inference is a one-off cost, you can afford to be lavish — long chain-of-thought, self-consistency voting across several samples, multi-step verification. Every point of teacher quality flows to the student.
Check licensing early. Some commercial model terms restrict using outputs to train competing models. Open-weight teachers with permissive licences avoid the question.
Step 2: Cover the input distribution
The student will only be good on inputs that resemble its training data. The most common distillation failure is a training set that is too clean, too short or too homogeneous compared to production traffic.
- Start from real production inputs where you can, sampled across segments, time and difficulty.
- Augment with synthetic inputs targeted at gaps: rare categories, long documents, multilingual cases, adversarial phrasing.
- Track coverage against a taxonomy of input types so you know what is underrepresented.
Typical dataset sizes: 5,000–50,000 examples for extraction and classification tasks; 20,000–200,000 for open-ended generation or reasoning.
Step 3: Filter ruthlessly
Teachers make mistakes, and a student trained on unfiltered outputs learns them faithfully. Filtering is where good distillation projects separate from mediocre ones.
- Programmatic checks: schema validity, executable code, consistency with source documents.
- Agreement filtering: keep examples where multiple teacher samples agree; route disagreements to human review.
- Calibrated judges: an LLM judge scoring each output against a rubric, validated against expert ratings.
- Deduplication and decontamination: remove near-duplicates and anything resembling the evaluation set.
It is normal to discard 15–40% of teacher outputs. The discarded examples are valuable too: their inputs mark the hard cases worth sending to human experts.
Step 4: Train the student
Run a bake-off across two to four candidate student sizes and families. Start with SFT on the filtered data. Then consider:
- On-policy distillation: the student generates, the teacher corrects or scores, and the student trains on its own outputs. This addresses exposure bias — the gap between imitating perfect demonstrations and recovering from one’s own errors.
- Logit distillation: where you have access to teacher token probabilities (typically open-weight teachers), training on the full distribution transfers more information per example.
- Preference tuning on disagreements: collect cases where student and teacher differ, have experts judge, and run DPO. This is how students come to exceed their teachers on the target task.
Step 5: Evaluate parity, then go further
Measure the student against the teacher on a frozen held-out set, sliced by input type. Report parity (agreement with teacher), accuracy against human ground truth where available, and cost and latency on target hardware. Look specifically at the long tail: aggregate parity of 98% can hide a category where the student fails half the time.
Then quantize — INT4 or FP8 — and re-run the eval. Well-distilled students usually lose less than a point. For the evaluation methodology, see how to evaluate a fine-tuned LLM.
Sizing the student
Choosing the student size is a trade-off between quality, cost and where it must run. Our rules of thumb, which the bake-off then confirms or overturns:
| Task | Typical student | Serving footprint |
|---|---|---|
| Classification, routing, intent detection | 0.5–3B | CPU or a fraction of one GPU |
| Field extraction from documents | 3–8B | One mid-range GPU, quantized |
| Templated summarization, drafting | 7–14B | One GPU, quantized |
| Tool-calling agents | 8–32B | One to two GPUs |
| Multi-step reasoning | 14–70B | Two to four GPUs |
Start one size smaller than you think you need. Distillation plus preference tuning regularly lets a smaller student reach parity, and every halving of model size roughly halves serving cost.
A worked example: customs classification
A global logistics company classified shipments into harmonized tariff codes using a frontier model with a long prompt and retrieval over tariff schedules. It worked — 88% accuracy — but cost over $60,000 a month and needed to run at warehouse sites with unreliable connectivity.
Teacher. We upgraded the teacher first: added structured reasoning through the tariff hierarchy, self-consistency across five samples, and retrieval of jurisdiction-specific notes. Teacher accuracy rose to 92%.
Data. 140,000 historical shipment descriptions across 41 jurisdictions, plus 30,000 synthetic descriptions targeting rare chapters. After agreement filtering and verifier checks against the tariff schedule, 118,000 examples remained. Disagreements went to licensed customs brokers.
Student. A 3B model fine-tuned on the filtered set reached 88.1%; one round of DPO on 6,000 broker-judged disagreements took it to 89.6% — below the upgraded teacher, above the original production system. Quantized to INT4, it runs on edge servers in every warehouse at 31× lower cost than the API pipeline.
Pitfalls we see most often
- Distilling a weak teacher. Spend on the teacher first.
- Training on clean inputs only. Production is messy; your data should be too.
- Skipping filtering. The student will memorize teacher errors with confidence.
- Measuring aggregate parity only. Slice the eval.
- Ignoring licences. Clear the teacher’s terms with counsel before generating a single example.
Our distillation service runs this process end to end in three to six weeks.