Rethinking Knowledge Distillation via Cross-Entropy. (arXiv:2208.10139v1 [cs.CV])

Knowledge Distillation (KD) has developed extensively and boosted various
tasks. The classical KD method adds the KD loss to the original cross-entropy
(CE) loss. We try to decompose the KD loss to explore its relation with the CE
loss. Surprisingly, we find it can be regarded as a combination of the CE loss
and an extra loss which has the identical form as the CE loss. However, we
notice the extra loss forces the student's relative probability to learn the …

arxiv cross-entropy cv distillation entropy knowledge