Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism

arXiv:2405.00739v1 Announce Type: new
Abstract: Does Knowledge Distillation (KD) really work? Conventional wisdom viewed it as a knowledge transfer procedure where a perfect mimicry of the student to its teacher is desired. However, paradoxical studies indicate that closely replicating the teacher's behavior does not consistently improve student generalization, posing questions on its possible causes. Confronted with this gap, we hypothesize that diverse attentions in teachers contribute to better student generalization at the expense of reduced fidelity in ensemble KD setups. …

abstract arxiv attention behavior cs.cv cs.lg distillation eess.iv fidelity however knowledge mimicry studies transfer type work