arxiv:2502.08606

Distillation Scaling Laws

Published on Feb 12

· Submitted by

akhaliq on Feb 13

Upvote

Authors:

Dan Busbridge ,

Abstract

We provide a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings reduce the risks associated with using distillation at scale; compute allocation for both the teacher and student models can now be done to maximize student performance. We provide compute optimal distillation recipes for when 1) a teacher exists, or 2) a teacher needs training. If many students are to be distilled, or a teacher already exists, distillation outperforms supervised pretraining until a compute level which grows predictably with student size. If one student is to be distilled and a teacher also needs training, supervised learning should be done instead. Additionally, we provide insights across our large scale study of distillation, which increase our understanding of distillation and inform experimental design.

View arXiv page View PDF Add to collection

Community

akhaliq

Paper submitter 3 days ago

jontitor

2 days ago

Scaling laws provide a valuable framework for optimizing teacher-student compute allocation, but real-world AI deployment rarely follows a fixed equation. In areas like chatbots, search engines, and recommendation systems, models don’t just train once and remain static—they evolve alongside shifting user behaviors, changing datasets, and emerging tasks. This study offers strong insights into when distillation is more efficient than supervised learning, but should we be thinking beyond one-time optimization? Could an adaptive scaling law, one that dynamically adjusts teacher-student relationships based on real-world performance and resource constraints, be a more effective approach for AI systems that operate in constantly changing environments? Additionally, the study assumes access to large-scale training data, but many low-resource languages and specialized technical fields lack the volume necessary for traditional scaling laws to apply. Could distillation strategies be refined not just for compute efficiency but also to prioritize knowledge transfer in data-scarce settings, making AI more adaptable and inclusive across different domains?