Toward Student-Oriented Teacher Network Training For Knowledge Distillation

arXiv:2206.06661v2 Announce Type: replace
Abstract: How to conduct teacher training for knowledge distillation is still an open problem. It has been widely observed that a best-performing teacher does not necessarily yield the best-performing student, suggesting a fundamental discrepancy between the current teacher training practice and the ideal teacher training strategy. To fill this gap, we explore the feasibility of training a teacher that is oriented toward student performance with empirical risk minimization (ERM). Our analyses are inspired by the recent …

abstract arxiv cs.ai cs.lg current distillation fundamental knowledge network network training practice strategy training type