Definition#
Knowledge distillation (Hinton et al. 2015) trains a student on the teacher’s output distribution rather than only on hard labels. With teacher logits , student logits and temperature , the classic objective mixes a soft-target term with the usual loss:
The factor keeps gradient magnitudes comparable across temperatures. For LLMs, teacher logits are often unavailable (API models), so enterprises use sequence-level distillation: generate teacher outputs, filter them with verifiers and judges, and fine-tune the student on the survivors.
Practical guide: LLM distillation; service: model distillation.