CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation. (arXiv:2209.07606v1 [cs.CV])

Knowledge distillation (KD) is an effective tool for compressing deep
classification models for edge devices. However, the performance of KD is
affected by the large capacity gap between the teacher and student networks.
Recent methods have resorted to a multiple teacher assistant (TA) setting for
KD, which sequentially decreases the size of the teacher model to relatively
bridge the size gap between these models. This paper proposes a new technique
called Curriculum Expert Selection for Knowledge Distillation (CES-KD) to
efficiently …

arxiv ces curriculum distillation expert knowledge