K-Culture GlossaryㅈTechnical words in the news
Distillation
A technique for transferring a large model's abilities into a smaller one — a "teacher model → student model" process.
In plain words
This technique works by generating a large volume of answers from a big, expensive "teacher model" and training a smaller, cheaper "student model" to imitate them. The name comes from the idea of "distilling" knowledge and pouring it into a new vessel.
Many of the compact models that run on phones are built this way. It's also a term at the center of industry conflict — companies have accused each other of secretly harvesting large volumes of answers from a rival's frontier model to train their own student models. A "distillation controversy" headline essentially means an accusation of "copying to learn."
Distillation vs. Quantization: What's the Difference?
Training a new, smaller model, Lowering the precision of an existing model
Both get described as 'making a model lighter,' so they look like the same technique. But one means training a separate, smaller model to learn from a larger one's abilities, while the other means shrinking the size of an existing model by coarsening its numbers. They're often combined too — distill down to a small model, then quantize it further.
| Aspect | Distillation | Quantization |
|---|---|---|
| What it does | Trains a smaller model using a larger model as a teacher | Reduces the number of digits used to represent the model's numbers |
| The result | A new model, distinct from the original | A lighter version of the same model |
| What it costs | Requires training — time and hardware | Just a conversion — much faster |
| What's lost | Falls short of the teacher's ability | Precision is trimmed, slightly blurring answers |
| In articles | "A small model distilled from the large model is released" | "A 4-bit version that runs on phones" |
Rule of thumbIf a new model is created, it's distillation; if the same model becomes lighter, it's quantization.
See also
Stories using this term
No story has used this term yet. New ones attach here automatically.