Knowledge Distillation — AI Glossary

Reference details

Related terms

• Model Pruning
• Model Distillation
• Model Soup

How it works

1. The teacher's full output distribution is computed for each training input, and the student is trained to match it rather than to match a single correct label.
2. The objective is a Kullback-Leibler divergence, and its asymmetry matters: it penalises the student for putting little mass where the teacher put much, far more than the reverse, so the student is pushed to cover every mode the teacher had.
3. Gradients from that term are rescaled by the square of the temperature, because raising the temperature shrinks the gradient magnitude and the correction keeps the term's weight stable as the temperature changes.
4. It is usually combined with ordinary cross-entropy against the hard label at some mixing weight, so the student is anchored to the ground truth as well as to the teacher's opinion.
5. The two distributions must be over the same vocabulary for the divergence to be defined at all, so classical distillation requires the student to share the teacher's tokenizer.
6. That constraint is why distilling across model families means generating text and fine-tuning on it instead — a different procedure that transfers only what the samples happen to contain.

Example: What the soft targets carry that a label does not

A hard label says the answer is “cat”. The teacher's distribution says cat 0.70, dog 0.25, fox 0.04, car 0.01. The extra information is in the ordering of the wrong answers: the teacher has encoded that this image is dog-like and not at all car-like, which is a similarity structure the one-hot label deletes entirely. The student trained on the distribution learns that structure and reaches accuracy it could not reach from labels alone with the same parameters. Raising the softmax temperature spreads the distribution further and makes those small differences count more.

Commonly confused with

Classical distillation trains on the teacher's output distribution and needs logit access. What is usually called distillation for LLMs today is different — generating text from a strong model and fine-tuning on it, which is supervised fine-tuning on synthetic data. It works, but it transfers only what the samples contain, and provider terms often restrict training competing models on their outputs.

When to use it

Reach for it when:

• Compressing a capable model into one you can afford to serve at volume
• Specialising a small model to a narrow task where the teacher is overqualified
• Where inference cost dominates and a few points of quality are an acceptable trade

Reach for something else when:

• Expecting the student to generalise beyond the distribution it was distilled on
• Without checking the teacher's terms of service for output-training restrictions
• As a substitute for better data: distilling a flawed teacher reproduces the flaws

Primary source

• Hinton et al., Distilling the Knowledge in a Neural Network (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Knowledge Distillation — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path