What Does Distillation Mean?
Distillation is a way of training a smaller AI model by having it learn from a larger one. The large model, called the teacher, produces answers or probability scores for many examples. The small model, called the student, is trained to match them. Because the teacher shows not just the right answer but how confident it is across alternatives, the student can learn more from each example than from raw data alone. The result is usually a model that is faster and cheaper to run and keeps much, but not all, of the teacher quality, especially on the tasks it was distilled for.
Why Does Distillation Matter?
Distillation is one reason small and mid-size models improved so quickly: many are trained partly on outputs from bigger models. For builders, it explains why a small model can be good enough for a narrow job like classification or extraction at a much lower price. It also has a legal side: many model providers forbid using their outputs to train competing models, so check the terms before distilling.
What Does Distillation Look Like in Practice?
A team uses a large model to label 50,000 support tickets by category, checks a sample by hand, then fine-tunes a small open-weight model on those labels. The small model runs on their own server for a fraction of the cost per ticket and handles the routine cases, while unusual tickets still go to the large model. They re-check a sample of the small model labels every month to catch drift.
What Do People Get Wrong About Distillation?
A distilled model is not a compressed copy of the big one. It learns to imitate behavior on the examples it saw, so it can be close on familiar tasks and noticeably weaker on anything outside them.
Which Terms Are Related?
- Fine-tuning: Extra training on your own examples that adjusts a model weights so it handles a specific task or style better.
- Open-weight model: A model whose trained weights you can download and run on your own hardware, under the terms of its license.
- Quantization: Storing model weights with fewer bits so a model needs less memory and runs faster, with some loss of quality.
- Frontier model: One of the most capable AI models at a given moment, typically closed and served by a large lab through an API.
- Inference: Running an already trained model to produce output. Every reply you get from an AI is one inference call.
All 28 terms are in the AI glossary. For models side by side, see frontier vs open-weight.
Frequently Asked Questions
Is distillation the same as fine-tuning?
Distillation usually uses fine-tuning as the training step, but the defining feature is where the training data comes from: a larger teacher model. Ordinary fine-tuning can use any data, such as human-written examples.
Can I distill from a commercial API model?
Check the provider terms first. Many commercial AI providers prohibit using their outputs to develop competing models. Some open-weight licenses allow it, and some restrict it. Read the license rather than assuming.
Are distilled models worse?
They are usually somewhat weaker than their teacher overall, especially on broad or unusual tasks, but they can be close on the tasks they were trained for, and they are much cheaper and faster to run.