Knowledge Distillation

Knowledge distillation is a model compression technique where a smaller 'student' model is trained to mimic the behaviour of a larger 'teacher' model. Rather than training the student model on raw data alone, it learns from the teacher's outputs — including the probability distributions over possible answers, which contain richer information than simple labels. The key insight is that the teacher model's 'soft labels' (probability distributions over all possible outputs) contain more information than 'hard labels' (the single correct answer). For instance, when classifying an image of a cat, the teacher might output 80% cat, 15% dog, 5% other — revealing that cats and dogs are visually similar. This nuanced information helps the student learn more effectively. Distillation can be applied at various levels: output-level (matching final predictions), feature-level (matching internal representations), and attention-level (matching attention patterns). It can also be combined with other compression techniques like quantisation and pruning for even greater efficiency gains.

Distillation enables businesses to deploy high-quality AI models in resource-constrained environments. A distilled model might achieve 95% of the quality of a large model while being 10x faster and 50x cheaper to run. This makes AI viable for edge devices, real-time applications, and high-volume services. The economics are compelling. Running a large language model for every customer query can be prohibitively expensive at scale. A distilled model trained to handle common queries can serve the majority of requests at a fraction of the cost, with the larger model reserved for complex cases. Distillation is also used by AI providers to create model families at different price and capability points. Understanding this helps businesses select the right model tier — using distilled models for routine tasks and larger models for tasks requiring maximum capability.