LumiNet: The Bright Side of Perceptual Knowledge Distillation

arXiv:2310.03669v2 Announce Type: replace
Abstract: In knowledge distillation literature, feature-based methods have dominated due to their ability to effectively tap into extensive teacher models. In contrast, logit-based approaches, which aim to distill `dark knowledge' from teachers, typically exhibit inferior performance compared to feature-based methods. To bridge this gap, we present LumiNet, a novel knowledge distillation algorithm designed to enhance logit-based distillation. We introduce the concept of 'perception', aiming to calibrate logits based on the model's representation capability. This concept addresses …

abstract aim arxiv bridge contrast cs.cv distillation feature gap knowledge literature performance teachers type