Model Distillation

Model Distillation is a training technique where small models learn from large model outputs. Core idea: rather than having the student model learn from human-labeled training data from scratch, have it learn "the probability distribution the teacher model outputs for each input." Why is this more effective? Consider a classification task with a cat as input. Direct supervised learning's training signal is "answer = cat" (hard label); distillation's training signal is the teacher's full output: "cat: 85%, kitten: 8%, dog: 4%..." The distillation signal doesn't just say "the answer is cat" — it implicitly conveys that "cat and kitten are conceptually close, cat and dog are somewhat similar" — these conceptual relationships let the student model acquire richer knowledge with far less training data. In the LLM domain, lightweight models like Claude Haiku acquire some capabilities through distillation from Claude Sonnet or Opus — Haiku "observes" Sonnet/Opus responses to various tasks, learning how to produce similar outputs in a lightweight form.

Distillation's core advantage lies in the information density provided by "soft targets." A teacher model's probability distribution for some input contains far richer knowledge than any single correct answer. Semantic relationships like "King - Man + Woman ≈ Queen" are implicitly encoded in these soft target probability distributions. Task-specific distillation is the most common industrial application: you don't need to distill all the teacher's capabilities — only capabilities for specific tasks. For example, if your application only needs sentiment analysis, you can use only teacher model outputs on sentiment analysis tasks to train a student model, producing an extremely lean model that excels at sentiment analysis with millisecond-level inference latency. Distillation data quality directly determines the student model's ceiling: using Claude Opus as teacher typically produces better students than using Claude Haiku — because stronger teachers provide richer "soft knowledge."

Model distillation's impact on your Claude usage is mainly about understanding why different Claude versions have this capability distribution: Haiku excels on certain tasks but is noticeably weaker than Sonnet on others — not entirely "Haiku's insufficient capability" but different knowledge transfer efficiency across task types during distillation. General tasks (translation, summarization, simple Q&A) distill very well; complex reasoning tasks (multi-step logic, difficult code) have higher distillation loss. For developers, if your application has very specific task requirements with cost and latency constraints, consider using high-quality outputs from Claude Opus or Sonnet to distill-train a lightweight model specialized for your task — this is many production AI applications' actual approach. Note API Terms of Service: using Claude outputs to train models for your own specific business tasks is generally permitted, but not for training models that directly compete with Anthropic. Confirm the latest terms before proceeding.

If you want to try distillation training using Claude outputs, practical recommendations: **Generate high-quality distillation data**: ensure teacher model (Claude Opus/Sonnet) output quality is sufficiently high — distillation data quality directly determines the student model's ceiling. Include diverse inputs (don't use just one question type) to help the student model develop better generalization. **Choose the right student architecture**: common choices are BERT-based (suited for classification, NER) or GPT-based small models (suited for generation tasks). For edge device deployment, consider lightweight base models like DistilBERT or Phi-3-mini. **Use standard distillation frameworks**: HuggingFace's `trl` library supports SFT (supervised fine-tuning) and KD (knowledge distillation) — the most mature open-source choice currently. OpenAI also provides a distillation API for directly training GPT-4o-mini on GPT-4 outputs — same principle.