Logit Standardization in Knowledge Distillation

arXiv:2403.01427v1 Announce Type: new
Abstract: Knowledge distillation involves transferring soft labels from a teacher to a student using a shared temperature-based softmax function. However, the assumption of a shared temperature between teacher and student implies a mandatory exact match between their logits in terms of logit range and variance. This side-effect limits the performance of student, considering the capacity discrepancy between them and the finding that the innate logit relations of teacher are sufficient for student to learn. To address …

abstract arxiv cs.cv distillation function knowledge labels match performance softmax standardization terms type variance