Contents
"A huge, high-performance AI is smart — but heavy and expensive." The technique that solves this is model distillation (knowledge distillation). By transferring the knowledge of a large "teacher" model to a small "student" model, you can keep 95%+ of the teacher's performance at one-tenth the size and speed — the best of both worlds.
This article explains how distillation works with a teacher-student analogy for beginners, and covers the benefits, the two approaches, and how it differs from fine-tuning and quantization. It then digs — without hype — into the "legal and terms-of-service issues" around distillation that drew major attention in 2026 (the OpenAI v. DeepSeek dispute and anti-distillation clauses).
Move a big teacher's knowledge to a small student
— Keep 95%+ of the performance at one-tenth the size
* Figures and examples in this article are quoted from public materials and news reports (as of June 2026). The legal points are general orientation; check experts and official sources for any specific case.
1. What is model distillation? A teacher-student analogy
Model distillation is a technique where a small "student" model is trained to reproduce the behavior of a large, high-performance "teacher" model. By mimicking the teacher's outputs, the student gains near-teacher ability at a far smaller size. As a real example, GPT-4o mini is described as distilled from GPT-4o.
The key is "soft labels": ordinary training only teaches "the answer is cat" (a hard label), but distillation passes the teacher's full probability distribution like "90% cat, 8% dog, 2% fox" to the student. That "degree of hesitation" carries rich information that the answer alone can't convey. A parameter called temperature then "softens" the probabilities so even subtle relationships between similar classes become visible.
By human analogy, a veteran (teacher) teaches a newcomer (student) not just "this is a cat" but the nuance of judgment — "a cat, though it's a borderline case with dog." So the student learns more deeply and efficiently than by rote. If you know how LLMs work, it's clear why a probability distribution is so information-rich.
2. Why distill? The benefits
The goal of distillation is simple — "keep as much smartness as possible while making it lighter, faster, and cheaper." The concrete benefits:
⚡ Fast and cheap
Less compute means lower latency and lower cost. It pays off in high-volume production.
📦 ~10x more compact
Reports show one-tenth the size while keeping 95%+ of performance.
📱 Runs on the edge
Easy to run even in resource-limited environments like phones and devices.
🎯 Strong for specialization
Easy to build small but accurate task-specific models.
In short, distillation is a bridge that brings "flagship-level smartness" down to "a cost you can run in production." For high-call-volume uses like agents, the cost difference compounds, so the value is especially large.
3. Two approaches: white-box / black-box
Distillation splits into two, by how much access you have to the teacher's "internals." This is directly tied to the legal point later.
White-box distillation
When you have full access to the teacher's weights and internal representations. The student learns not only outputs but the internal decision process, so the transfer goes deeper. Usable when your own model or an OSS model is the teacher.
Black-box distillation
When you only see the teacher's outputs (API responses). You collect input-output pairs and train the student on them. Using another company's API as the teacher can violate its terms (see below).
4. vs quantization and fine-tuning
Distillation is easily confused with similar "make a model lighter/different" techniques — quantization and fine-tuning. Since their goals differ, let's sort them out.
| Technique | What it does | Goal |
|---|---|---|
| Distillation | Train a separate small model on a big model's knowledge | Small and fast, while keeping performance |
| Quantization | Compress the same model by lowering weight precision | Save memory/speed (same model inside) |
| Fine-tuning | Further-train an existing model for a specific task | Adapt to a use case/domain (size roughly unchanged) |
Roughly: distillation = "move the wisdom into a different, smaller vessel," quantization = "make the same vessel lighter," fine-tuning = "add domain knowledge to the same vessel." The three aren't mutually exclusive — they're often combined (e.g., quantize a distilled small model further).
5. The legal and terms-of-service reality
This is the part that became a big issue in 2026. The technique of distillation is entirely legitimate. What becomes a problem is "whose outputs you use, and for what."
The crux: the terms of use of OpenAI, Anthropic, Mistral, xAI, and others include an "anti-competitive distillation" clause prohibiting using their service's outputs to develop a competing model. So distilling a competing model using the outputs of a restricted API can violate the terms — even though it's technically possible.
This escalated into a real dispute in the OpenAI v. DeepSeek case. According to reports, OpenAI alleged that "accounts believed to be linked to DeepSeek circumvented access restrictions to obtain model outputs and used them for distillation" (early 2026). Meanwhile, DeepSeek's own terms of use reportedly permit using its service's outputs to train other models (including distillation). The point is that the assessment changes depending on "whose API terms apply."
This issue casts a shadow over the latest models too. With Claude Fable 5 / Mythos 5, a design was reported in which safety classifiers restrict responses on work flagged as "model distillation." The tension around distillation continues on both the regulatory and vendor-policy fronts. In practice, the rule is to always check the terms of use of the teacher model you use.
Tips for distilling safely
- Use your own model or a licensed OSS model as the teacher (many permit distillation)
- Before using another company's commercial API as the teacher, check its anti-distillation clause
- Carefully judge whether the use amounts to "developing a competing model"
Summary
Model distillation is a powerful technique that moves a big AI's smartness into a small AI and brings it down to a cost you can run in production. Let's recap.
Key takeaways
- 🧑🏫 Teacher → student: move a big model's knowledge to a small one. Soft labels + temperature are the key.
- ⚡ ~10x smaller and faster, keeping 95%+ of performance. Great for edge and low-cost ops.
- 🔓 Two approaches: white-box (sees internals) / black-box (outputs only).
- 🔀 Different from quantization and fine-tuning: move vessels / lighten / add domain knowledge.
- ⚖️ Mind the terms: the technique is legitimate, but using a restricted API's outputs to build a competitor can violate ToS.
"Smartness from the big model, operation from the small model." Distillation makes that combination possible. But who you choose as the teacher changes the outcome both technically and legally. For the basics, see what an LLM is; for a related technique, fine-tuning.
FAQ
Q. How much performance is lost by distilling?
A. It depends on the use case, but reports say a well-designed distillation can "keep 95%+ of performance at one-tenth the size." It's not identical, so always confirm it's within tolerance via evaluation.
Q. When do I use distillation vs quantization?
A. Distillation "moves knowledge into a separate, smaller model"; quantization "compresses the same model's weights." Their goals differ, so they aren't exclusive — combining them (e.g., quantize a distilled small model) is common.
Q. Can I use another AI's outputs to build my own model?
A. It depends on that provider's terms. OpenAI, Anthropic, and others have anti-distillation clauses prohibiting using outputs to develop competing models. It can violate the terms even if technically possible, so always check the terms of the service you use as the teacher.
Q. Can a beginner do distillation?
A. The concept is simple, but implementation needs machine-learning knowledge. Start with understanding the mechanism. Cloud providers (e.g., Azure) also offer services that assist distillation, so there are easier options than building from scratch.
