Rethinking Centered Kernel Alignment in Knowledge Distillation

arXiv:2401.11824v2 Announce Type: replace
Abstract: Knowledge distillation has emerged as a highly effective method for bridging the representation discrepancy between large-scale models and lightweight models. Prevalent approaches involve leveraging appropriate metrics to minimize the divergence or distance between the knowledge extracted from the teacher model and the knowledge learned by the student model. Centered Kernel Alignment (CKA) is widely used to measure representation similarity and has been applied in several knowledge distillation methods. However, these methods are complex and fail …

abstract alignment arxiv cs.cv distillation divergence kernel knowledge large-scale models metrics representation scale type