Knowledge distillation trains a small 'student' model to replicate the behavior of a large 'teacher' model, producing a separate, deployable model rather than a compressed version. The key mechanism is soft labels — the teacher's full probability distribution over outputs — which carry richer information than hard labels, enabling the student to learn relational structure between categories. Three main methods exist: output distillation (matching final probabilities), feature distillation (matching internal representations), and synthetic data distillation (fine-tuning on teacher-generated examples), with the last being most common due to API-only access to many models. Real-world results like DeepSeek's 7B model outperforming a 32B model on math benchmarks show strong but narrow gains. Key limits include the teacher setting a performance ceiling, capacity gaps between teacher and student hurting transfer, architecture mattering more than size, and unintended traits being passed from teacher to student. The newest direction automates the entire distillation loop — data generation, fine-tuning, and evaluation — with the teacher model driving the process end-to-end.
