CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective

arXiv:2404.14109v1 Announce Type: new
Abstract: In this paper, we present a simple yet effective contrastive knowledge distillation approach, which can be formulated as a sample-wise alignment problem with intra- and inter-sample constraints. Unlike traditional knowledge distillation methods that concentrate on maximizing feature similarities or preserving class-wise semantic correlations between teacher and student features, our method attempts to recover the "dark knowledge" by aligning sample-wise teacher and student logits. Specifically, our method first minimizes logit differences within the same sample by …

arxiv cs.cv distillation knowledge perspective sample type wise