What Is Model Distillation? Moving Knowledge From a Big AI to a Small One
A huge, high-performance AI is smart but heavy and expensive; model distillation (knowledge distillation) solves this by transferring a large teacher model's knowledge to a small student model, keeping 95%+ of the teacher's performance at one-tenth the size and speed. This article explains it with a teacher-student analogy. The key is soft labels: ordinary training teaches only "the answer is cat" (hard label), while distillation passes the teacher's full probability distribution like "90% cat, 8% dog, 2% fox," whose degree of hesitation carries rich information; a temperature parameter softens the probabilities to reveal subtle relationships (real example: GPT-4o mini distilled from GPT-4o). Benefits: fast and cheap, ~10x more compact while keeping 95%+ performance, runs on the edge, strong for specialization. Two approaches: white-box (full access to weights and internal representations, deeper transfer; for your own or OSS models) and black-box (only outputs/API responses visible; using another company's API as teacher can violate terms). It differs from quantization (compress the same model's weight precision) and fine-tuning (further-train an existing model for a task) — distillation moves knowledge into a separate small model, and the three are combinable. The legal/ToS reality was a big 2026 issue: the technique is legitimate, but OpenAI, Anthropic, Mistral, and xAI include anti-competitive distillation clauses prohibiting using outputs to build competing models, so distilling a competitor from a restricted API can violate terms. The OpenAI v. DeepSeek dispute (OpenAI alleged DeepSeek-linked accounts circumvented restrictions to obtain outputs for distillation, while DeepSeek's terms reportedly permit distilling its outputs) shows the assessment depends on whose API terms apply, and Claude Fable 5/Mythos 5 reportedly restrict responses on distillation-flagged work. Tips: use your own or licensed OSS models as teacher, check anti-distillation clauses before using a commercial API, and judge whether the use is "developing a competing model." Smartness from the big model, operation from the small — but who you pick as teacher changes the outcome technically and legally. Figures are quoted from public materials, directional.