The Evolution of Knowledge Distillation: A Survey of Advanced Teacher-Student Training Paradigms
Introduction: Beyond Classical Knowledge Distillation Knowledge Distillation (KD) has emerged as a cornerstone technique in machine learning, fundamentally addressing the tension between model performance and deployment efficiency.1 As deep neural Read More …